tokenize-rt

What it is and what it does

tokenize-rt is a wrapper around Python's standard tokenize module that solves the roundtripping problem: the stdlib tokenizer discards information (whitespace, escaped newlines) that makes it impossible to reconstruct the original source exactly. This library adds two new token types—ESCAPED_NL for backslash-escaped newlines and UNIMPORTANT_WS for whitespace—and provides a Token data type with position metadata. It's designed for developers building refactoring and code-rewriting tools that need to parse Python source, modify it, and write it back without losing formatting or comments.

The package provides src_to_tokens() to parse source into a list of Token objects and tokens_to_src() to reconstruct the original source from those tokens. It also normalizes string prefixes and Python 2 literals across Python versions, making it easier to write cross-version refactoring tools. Helper functions like reversed_enumerate and rfind_string_parts support common rewriting patterns.

Use it for:

• Building automated code refactoring tools that preserve formatting and comments.
• Implementing linters or code transformers that need to modify source while maintaining roundtrip fidelity.
• Cross-referencing AST nodes with their exact token positions and source text.
• Rewriting Python 2 code while running on Python 3 with normalized token representations.
• Parsing and manipulating string literals and their prefixes in source code.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Wraps Python's stdlib tokenize module to enable proper roundtripping of source code, adding ESCAPED_NL and UNIMPORTANT_WS tokens and a Token data type for lossless tokenization.

Yes. tokenize-rt solves a genuine limitation of the stdlib tokenizer for any tool that needs to parse and rewrite Python source code. It has no dependencies, low install friction, active maintenance, a permissive MIT license, and no known vulnerabilities. Install it if you're building a refactoring tool, linter, or code transformer.

Install

tokenize-rt on PyPI

pip install tokenize-rt

uv add tokenize-rt

poetry add tokenize-rt

Installing tokenize-rt

Before you install

Low install friction with no runtime dependencies. Active maintenance as of 2025-05-23, with recent commits and a stable repository.

License in practice

MIT license permits unrestricted use, modification, and distribution in both open-source and proprietary projects.

Quickstart

pip install tokenize-rt

from tokenize_rt import src_to_tokens, tokens_to_src

tokens = src_to_tokens('x = 1')
result = tokens_to_src(tokens)

Requires Python 3.9 or later.

Verify before relying

• Whether the package handles all edge cases in modern Python syntax (f-strings, walrus operators, match statements).
• Performance characteristics when tokenizing very large source files.

Package facts

Evidence: tokenize_rt-6.2.0-py2.py3-none-any.whl

Tags

More Software Development packages