{"categories":[{"label":"Software Development","url":"https://skillfed.io/packages/category/software-development/2"}],"enrichment":{"capability":"Wraps Python's stdlib tokenize module to enable proper roundtripping of source code, adding ESCAPED_NL and UNIMPORTANT_WS tokens and a Token data type for lossless tokenization.","skillfed_tags":["tokenization","code-rewriting","ast-tools"],"use_cases":["Building automated code refactoring tools that preserve formatting and comments.","Implementing linters or code transformers that need to modify source while maintaining roundtrip fidelity.","Cross-referencing AST nodes with their exact token positions and source text.","Rewriting Python 2 code while running on Python 3 with normalized token representations.","Parsing and manipulating string literals and their prefixes in source code."],"what_it_does":"tokenize-rt is a wrapper around Python's standard tokenize module that solves the roundtripping problem: the stdlib tokenizer discards information (whitespace, escaped newlines) that makes it impossible to reconstruct the original source exactly. This library adds two new token types\u2014ESCAPED_NL for backslash-escaped newlines and UNIMPORTANT_WS for whitespace\u2014and provides a Token data type with position metadata. It's designed for developers building refactoring and code-rewriting tools that need to parse Python source, modify it, and write it back without losing formatting or comments.\n\nThe package provides src_to_tokens() to parse source into a list of Token objects and tokens_to_src() to reconstruct the original source from those tokens. It also normalizes string prefixes and Python 2 literals across Python versions, making it easier to write cross-version refactoring tools. Helper functions like reversed_enumerate and rfind_string_parts support common rewriting patterns.","worth_installing":"Yes. tokenize-rt solves a genuine limitation of the stdlib tokenizer for any tool that needs to parse and rewrite Python source code. It has no dependencies, low install friction, active maintenance, a permissive MIT license, and no known vulnerabilities. Install it if you're building a refactoring tool, linter, or code transformer."},"id":"tokenize-rt","links":{"html":"https://skillfed.io/packages/tokenize-rt","md":"https://skillfed.io/packages/tokenize-rt.md","pypi":"https://pypi.org/project/tokenize-rt/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2025-05-23","license_spdx":null,"license_treatment":"permissive","name":"tokenize-rt","python_support":"supports_current","summary":"A wrapper around the stdlib `tokenize` which roundtrips."},"popularity":{"monthly_downloads":7740890,"position":1699,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"6.2.0"}