untokenize

What it is and what it does

untokenize is a small utility that reverses Python's tokenization process—taking a stream of tokens and reconstructing the original source code. The key difference from the standard library's tokenize.untokenize() is that it preserves the exact whitespace that appeared between tokens in the original source, rather than normalizing or collapsing it. This makes it useful when you need to round-trip code through tokenization without losing formatting details.

The package has no external dependencies and is straightforward to use: pass a token stream to untokenize.untokenize() and get back the reconstructed source. However, the project is abandoned—last updated in 2014—and has not been maintained or tested against modern Python versions. It was originally written for Python 2.6, 2.7, and 3, but its actual compatibility with current Python releases is unknown.

Use it for:

• Reconstructing source code after tokenization in code analysis or transformation tools that need to preserve formatting.
• Building code generators or refactoring tools that must round-trip code without altering whitespace.
• Testing tokenization round-trip fidelity—verifying that tokenize → untokenize produces byte-for-byte identical output.
• Implementing syntax-aware code editors or linters that need to preserve original formatting while making targeted changes.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Converts Python tokens back into source code while preserving original whitespace, unlike the standard library's tokenize.untokenize() which loses formatting.

No. The package is abandoned (last update 2014, last commit 2019-05-24) with no active maintenance or security updates. Compatibility with modern Python versions is unverified. Unless you have a specific legacy codebase or a very narrow use case that depends on this exact implementation, the standard library's tokenize module or a maintained alternative is a safer choice.

Install

untokenize on PyPI

pip install untokenize

uv add untokenize

poetry add untokenize

Installing untokenize

Before you install

High install friction with no runtime dependencies. The package is abandoned—last commit was 2019-05-24, over four years ago—and has not been updated since its 0.1.1 release in 2014. No active maintenance or security updates should be expected.

License in practice

Licensed under the Expat License (MIT), a permissive open-source license with minimal restrictions. You may use, modify, and distribute the package freely in proprietary or open-source projects.

Quickstart

import untokenize
import tokenize
import io

tokens = tokenize.generate_tokens(io.StringIO(source_code).readline)
reconstructed = untokenize.untokenize(tokens)

The package was written for Python 2.6, 2.7, and 3 but has not been maintained since 2014; compatibility with modern Python versions is unverified.

Verify before relying

• Whether the package works correctly with Python versions released after 2014.
• Whether the whitespace-preservation behavior handles all edge cases in modern Python syntax.
• Active alternatives or whether the standard library's tokenize module has since improved.

Package facts

Evidence: untokenize-0.1.1.tar.gz

Keywords: tokenize, untokenize, transform, generate

Tags

More Text Processing packages