tokenize-rt
A wrapper around the stdlib `tokenize` which roundtrips.
Decision gist · record as of 2026-08-14
Yes. tokenize-rt solves a genuine limitation of the stdlib tokenizer for any tool that needs to parse and rewrite Python source code. It has no dependencies, low install friction, active maintenance, a permissive MIT license, and no known vulnerabilities. Install it if you're building a refactoring tool, linter, or code transformer.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.9 or later.
- Low install friction with no runtime dependencies.
- Active maintenance as of 2025-05-23, with recent commits and a stable repository.
License · maintenance · safety
MIT (permissive) — MIT license permits unrestricted use, modification, and distribution in both open-source and proprietary projects.
last release 2025-05-23 (448 days) · last repo commit 2026-07-15 · 61 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 7,740,890 downloads/mo, #1,699 on PyPI
Alternatives
Verify before relying
pip install tokenize-rt
from tokenize_rt import src_to_tokens, tokens_to_src
tokens = src_to_tokens('x = 1')
result = tokens_to_src(tokens)- Whether the package handles all edge cases in modern Python syntax (f-strings, walrus operators, match statements).
- Performance characteristics when tokenizing very large source files.
What it is and what it does
tokenize-rt is a wrapper around Python's standard tokenize module that solves the roundtripping problem: the stdlib tokenizer discards information (whitespace, escaped newlines) that makes it impossible to reconstruct the original source exactly. This library adds two new token types—ESCAPED_NL for backslash-escaped newlines and UNIMPORTANT_WS for whitespace—and provides a Token data type with position metadata. It's designed for developers building refactoring and code-rewriting tools that need to parse Python source, modify it, and write it back without losing formatting or comments.
The package provides src_to_tokens() to parse source into a list of Token objects and tokens_to_src() to reconstruct the original source from those tokens. It also normalizes string prefixes and Python 2 literals across Python versions, making it easier to write cross-version refactoring tools. Helper functions like reversed_enumerate and rfind_string_parts support common rewriting patterns.
Use it for
- Building automated code refactoring tools that preserve formatting and comments.
- Implementing linters or code transformers that need to modify source while maintaining roundtrip fidelity.
- Cross-referencing AST nodes with their exact token positions and source text.
- Rewriting Python 2 code while running on Python 3 with normalized token representations.
- Parsing and manipulating string literals and their prefixes in source code.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
tokenize-rt solves a genuine limitation of the stdlib tokenizer for any tool that needs to parse and rewrite Python source code. It has no dependencies, low install friction, active maintenance, a permissive MIT license, and no known vulnerabilities. Install it if you're building a refactoring tool, linter, or code transformer.
Install
tokenize-rt on PyPI
Before you install
Low install friction with no runtime dependencies. Active maintenance as of 2025-05-23, with recent commits and a stable repository.
Requires Python 3.9 or later.
License in practice
MIT license permits unrestricted use, modification, and distribution in both open-source and proprietary projects.
Quickstart
pip install tokenize-rt
from tokenize_rt import src_to_tokens, tokens_to_src
tokens = src_to_tokens('x = 1')
result = tokens_to_src(tokens)Verify before relying
- Whether the package handles all edge cases in modern Python syntax (f-strings, walrus operators, match statements).
- Performance characteristics when tokenizing very large source files.
Package facts
| License | MIT permissive |
| Python support | Supports the current Python release >=3.9 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | None |
| Maintenance | Actively maintained 448 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 7,740,890 / month, #1,699 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Programming Language :: Python :: 3Programming Language :: Python :: 3 :: OnlyProgramming Language :: Python :: Implementation :: CPythonProgramming Language :: Python :: Implementation :: PyPy |
Evidence: tokenize_rt-6.2.0-py2.py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “python tokenize roundtrip”
- tokenize-rtWraps Python's stdlib tokenize module to enable proper roundtripping…
- json5kitParses JSON5 (JSON with comments, trailing commas, unquoted keys)…
- astunparseConverts Python abstract syntax trees (ASTs) back into source code…
Give your agent the search over MCP, or paste the wish link into any chat.
More Software Development packages
Provides backported and experimental type hints for Python 3.9+, allowing use of newer typing features on older Python versions and enabling early experimentation with type system PEPs before they enter the standard library.
NumPy provides an N-dimensional array object and a comprehensive suite of mathematical, linear algebra, Fourier transform, and random number functions for scientific computing in Python.
FastAPI is a Python web framework for building REST APIs using type hints, with automatic request validation, serialization, and interactive API documentation.
Provides a way to document function parameters, class attributes, return types, and variables inline using Python's `Annotated` type hint syntax instead of traditional docstrings.
Typer builds command-line applications from Python functions using type hints, automatically generating help text, argument parsing, and shell completion.
Install it if you are building CLIs in Python.
Distlib provides low-level packaging utilities for building, distributing, and managing Python software—including metadata handling, version specifiers, wheel support, script installation, and dependency resolution.
See also untokenize · google-pasta · asttokens · curated-tokenizers · ast-comments · match · future-fstrings · razdel · tensorflow-text · astunparse