{"categories":[{"label":"Text Processing","url":"https://skillfed.io/packages/category/text-processing/2"}],"enrichment":{"capability":"Converts Python tokens back into source code while preserving original whitespace, unlike the standard library's tokenize.untokenize() which loses formatting.","skillfed_tags":["tokenization","abandoned"],"use_cases":["Reconstructing source code after tokenization in code analysis or transformation tools that need to preserve formatting.","Building code generators or refactoring tools that must round-trip code without altering whitespace.","Testing tokenization round-trip fidelity\u2014verifying that tokenize \u2192 untokenize produces byte-for-byte identical output.","Implementing syntax-aware code editors or linters that need to preserve original formatting while making targeted changes."],"what_it_does":"untokenize is a small utility that reverses Python's tokenization process\u2014taking a stream of tokens and reconstructing the original source code. The key difference from the standard library's tokenize.untokenize() is that it preserves the exact whitespace that appeared between tokens in the original source, rather than normalizing or collapsing it. This makes it useful when you need to round-trip code through tokenization without losing formatting details.\n\nThe package has no external dependencies and is straightforward to use: pass a token stream to untokenize.untokenize() and get back the reconstructed source. However, the project is abandoned\u2014last updated in 2014\u2014and has not been maintained or tested against modern Python versions. It was originally written for Python 2.6, 2.7, and 3, but its actual compatibility with current Python releases is unknown.","worth_installing":"No. The package is abandoned (last update 2014, last commit 2019-05-24) with no active maintenance or security updates. Compatibility with modern Python versions is unverified. Unless you have a specific legacy codebase or a very narrow use case that depends on this exact implementation, the standard library's tokenize module or a maintained alternative is a safer choice."},"id":"untokenize","links":{"html":"https://skillfed.io/packages/untokenize","md":"https://skillfed.io/packages/untokenize.md","pypi":"https://pypi.org/project/untokenize/"},"maintenance":{"status":"abandoned"},"meta":{"latest_release":"2014-02-08","license_spdx":null,"license_treatment":"permissive","name":"untokenize","python_support":"unspecified","summary":"Transforms tokens into original source code (while preserving whitespace)."},"popularity":{"monthly_downloads":493860,"position":6353,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.1.1"}