---
id: tokenize-rt
version: "6.2.0"
license: MIT
license_treatment: permissive
maintenance: active
---

# tokenize-rt — A wrapper around the stdlib `tokenize` which roundtrips.

License: permissive · Maintenance: active · Downloads: 7.7M/mo

## What it is and what it does

tokenize-rt is a wrapper around Python's standard tokenize module that solves the roundtripping problem: the stdlib tokenizer discards information (whitespace, escaped newlines) that makes it impossible to reconstruct the original source exactly. This library adds two new token types—ESCAPED_NL for backslash-escaped newlines and UNIMPORTANT_WS for whitespace—and provides a Token data type with position metadata. It's designed for developers building refactoring and code-rewriting tools that need to parse Python source, modify it, and write it back without losing formatting or comments.

The package provides src_to_tokens() to parse source into a list of Token objects and tokens_to_src() to reconstruct the original source from those tokens. It also normalizes string prefixes and Python 2 literals across Python versions, making it easier to write cross-version refactoring tools. Helper functions like reversed_enumerate and rfind_string_parts support common rewriting patterns.

Use it for:

- Building automated code refactoring tools that preserve formatting and comments.
- Implementing linters or code transformers that need to modify source while maintaining roundtrip fidelity.
- Cross-referencing AST nodes with their exact token positions and source text.
- Rewriting Python 2 code while running on Python 3 with normalized token representations.
- Parsing and manipulating string literals and their prefixes in source code.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Wraps Python's stdlib tokenize module to enable proper roundtripping of source code, adding ESCAPED_NL and UNIMPORTANT_WS tokens and a Token data type for lossless tokenization.

Yes. tokenize-rt solves a genuine limitation of the stdlib tokenizer for any tool that needs to parse and rewrite Python source code. It has no dependencies, low install friction, active maintenance, a permissive MIT license, and no known vulnerabilities. Install it if you're building a refactoring tool, linter, or code transformer.

## Install

pip install tokenize-rt
uv add tokenize-rt
poetry add tokenize-rt

## Installing tokenize-rt

Before you install: Low install friction with no runtime dependencies. Active maintenance as of 2025-05-23, with recent commits and a stable repository.

License in practice: MIT license permits unrestricted use, modification, and distribution in both open-source and proprietary projects.

Quickstart:

pip install tokenize-rt

from tokenize_rt import src_to_tokens, tokens_to_src

tokens = src_to_tokens('x = 1')
result = tokens_to_src(tokens)

Requires Python 3.9 or later.

Verify before relying:

- Whether the package handles all edge cases in modern Python syntax (f-strings, walrus operators, match statements).
- Performance characteristics when tokenizing very large source files.

## Package facts

- License: MIT (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 7.7M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

python tokenize roundtrip, source code tokenization, ast token mapping, python refactoring tools, preserve whitespace tokenize, token to source conversion, code rewriting tokenizer, tokenization, code-rewriting, ast-tools

[View on SkillFed](https://skillfed.io/packages/tokenize-rt) · [View on PyPI](https://pypi.org/project/tokenize-rt/)