untokenize
Transforms tokens into original source code (while preserving whitespace).
Decision gist · record as of 2026-08-14
No. The package is abandoned (last update 2014, last commit 2019-05-24) with no active maintenance or security updates. Compatibility with modern Python versions is unverified. Unless you have a specific legacy codebase or a very narrow use case that depends on this exact implementation, the standard library's tokenize module or a maintained alternative is a safer choice.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- The package was written for Python 2.6, 2.7, and 3 but has not been maintained since 2014; compatibility with modern Python versions is unverified.
- High install friction with no runtime dependencies.
- The package is abandoned—last commit was 2019-05-24, over four years ago—and has not been updated since its 0.1.1 release in 2014.
License · maintenance · safety
Expat License (permissive) — Licensed under the Expat License (MIT), a permissive open-source license with minimal restrictions. You may use, modify, and distribute the package freely in proprietary or open-source projects.
last release 2014-02-08 (4570 days) · last repo commit 2019-05-24 · 9 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 493,860 downloads/mo, #6,353 on PyPI
Alternatives
Verify before relying
import untokenize
import tokenize
import io
tokens = tokenize.generate_tokens(io.StringIO(source_code).readline)
reconstructed = untokenize.untokenize(tokens)- Whether the package works correctly with Python versions released after 2014.
- Whether the whitespace-preservation behavior handles all edge cases in modern Python syntax.
- Active alternatives or whether the standard library's tokenize module has since improved.
What it is and what it does
untokenize is a small utility that reverses Python's tokenization process—taking a stream of tokens and reconstructing the original source code. The key difference from the standard library's tokenize.untokenize() is that it preserves the exact whitespace that appeared between tokens in the original source, rather than normalizing or collapsing it. This makes it useful when you need to round-trip code through tokenization without losing formatting details.
The package has no external dependencies and is straightforward to use: pass a token stream to untokenize.untokenize() and get back the reconstructed source. However, the project is abandoned—last updated in 2014—and has not been maintained or tested against modern Python versions. It was originally written for Python 2.6, 2.7, and 3, but its actual compatibility with current Python releases is unknown.
Use it for
- Reconstructing source code after tokenization in code analysis or transformation tools that need to preserve formatting.
- Building code generators or refactoring tools that must round-trip code without altering whitespace.
- Testing tokenization round-trip fidelity—verifying that tokenize → untokenize produces byte-for-byte identical output.
- Implementing syntax-aware code editors or linters that need to preserve original formatting while making targeted changes.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
No.
The package is abandoned (last update 2014, last commit 2019-05-24) with no active maintenance or security updates. Compatibility with modern Python versions is unverified. Unless you have a specific legacy codebase or a very narrow use case that depends on this exact implementation, the standard library's tokenize module or a maintained alternative is a safer choice.
Install
untokenize on PyPI
Before you install
High install friction with no runtime dependencies. The package is abandoned—last commit was 2019-05-24, over four years ago—and has not been updated since its 0.1.1 release in 2014. No active maintenance or security updates should be expected.
The package was written for Python 2.6, 2.7, and 3 but has not been maintained since 2014; compatibility with modern Python versions is unverified.
License in practice
Licensed under the Expat License (MIT), a permissive open-source license with minimal restrictions. You may use, modify, and distribute the package freely in proprietary or open-source projects.
Quickstart
import untokenize
import tokenize
import io
tokens = tokenize.generate_tokens(io.StringIO(source_code).readline)
reconstructed = untokenize.untokenize(tokens)Verify before relying
- Whether the package works correctly with Python versions released after 2014.
- Whether the whitespace-preservation behavior handles all edge cases in modern Python syntax.
- Active alternatives or whether the standard library's tokenize module has since improved.
Package facts
| License | Expat License permissive |
| Python support | Not specified |
| Install friction | High. Source build required |
| Runtime dependencies | None |
| Maintenance | Abandoned 4,570 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 493,860 / month, #6,353 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Environment :: ConsoleIntended Audience :: DevelopersLicense :: OSI Approved :: MIT LicenseProgramming Language :: Python :: 2.6Programming Language :: Python :: 2.7Programming Language :: Python :: 3 |
Evidence: untokenize-0.1.1.tar.gz
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “tokenize reverse”
- untokenizeConverts Python tokens back into source code while preserving…
- tokenize-rtWraps Python's stdlib tokenize module to enable proper roundtripping…
- sacremosesSacremoses provides tokenization, detokenization, truecasing, and…
Give your agent the search over MCP, or paste the wish link into any chat.
More Text Processing packages
A drop-in replacement for Python's standard `re` module that adds advanced regex features like nested sets, fuzzy matching, lookaround in conditionals, and full Unicode case-folding while maintaining backward compatibility.
pyparsing provides a library for building text parsers directly in Python code using composable grammar classes, handling quoted strings, whitespace variation, and embedded comments without regex or lex/yacc.
Install it if you need to parse text or define grammars programmatically.
fonttools manipulates font files in multiple formats (TrueType, OpenType, AFM, Type 1, Mac-specific) and includes TTX, a tool to convert fonts to and from XML text format.
Install it if you need to read, write, or manipulate fonts programmatically or via the TTX command-line tool.
Docutils converts plaintext documentation in reStructuredText format into multiple output formats including HTML, XML, and LaTeX using a modular processing system.
RapidFuzz provides fast fuzzy string matching using Levenshtein Distance and related metrics, implemented mostly in C++ with Python bindings for rapid similarity scoring and approximate string matching.
Install it if you need fuzzy string matching; it's a solid replacement for FuzzyWuzzy with better licensing and performance.
tinycss2 parses CSS strings into token and block objects, and generates CSS strings from those objects, following the CSS Syntax Level 3 specification without enforcing specific properties or values.
Install it if your project requires CSS tokenization or syntax manipulation.
See also tokenize-rt · sacremoses · imperfect · tomlkit · tensorflow-text · tokenizers · semchunk · python-minifier · htmlmin2 · tokenizer