pytokens

A Fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.

pytokens Permissive license MIT License Copyright (c) 2024 Tushar Sadhwani Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to… (full text in the JSON record) Active ★ 4 v0.4.1 released 2026-01-30

Install

pytokens on PyPI

pip

pip install pytokens

uv

uv add pytokens

poetry

poetry add pytokens

Package facts

License	MIT License Copyright (c) 2024 Tushar Sadhwani Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to… (full text in the JSON record) (permissive)
Python support	supports the current Python release (>=3.8)
Install friction	low — pure-Python wheel
Runtime dependencies	none
Maintenance	actively maintained — 195 days since the last release
Last repo commit	2026-02-18
First released	2024-12-13
Popularity	one of the top 1,000 most-downloaded packages on PyPI (30-day window, as of 2026-08-13)
Known vulnerabilities	none known (OSV.dev, checked 2026-08-13)

Evidence: pytokens-0.4.1-py3-none-any.whl

License :: OSI Approved :: MIT LicenseOperating System :: OS IndependentProgramming Language :: Python :: 3Programming Language :: Python :: 3 :: OnlyProgramming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Programming Language :: Python :: 3.8Programming Language :: Python :: 3.9Programming Language :: Python :: Implementation :: CPythonTyping :: Typed

About pytokens

from the package's own PyPI description — quoted content, verbatim

pytokens

A Fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.

Installation

pip install pytokens

Usage

python -m pytokens path/to/file.py

Local Development / Testing

Create and activate a virtual environment
Run PYTOKENS_USE_MYPYC=0 pip install -e '.[dev]' to do an editable install
Run pytest to run tests

Type Checking

Run mypy .

Compilation with mypyc

By default, we compile with mypyc. Use the PYTOKENS_USE_MYPYC environment variable to control this.

To check if you are using a compiled version, see whether the output of this is .py or .so:

python -c "import pytokens; print(pytokens.__file__)"

Create and upload a package to PyPI

Make sure to bump the version in pyproject.toml.
Push to Github so CI can build the wheels and sdist.
Download the artifacts from the CI run.
Find the "Build wheels" job, click "Summary", scroll down to the bottom to see "Artifacts" and download the cibw-wheels artifact.
Unzip the artifact.
Upload the contents of the artifact to PyPI
e.g. via twine upload cibw-wheels/*

Read as markdown · JSON record · Source repository · Homepage

AI interpretation — verify before relying

AI-generated interpretation of the package facts above; every digit, version, license, or vulnerability id it cites is grounded in the facts already shown on this page

A fast, spec-compliant Python tokenizer that generates tokens for Python 3.14+ code while running on Python 3.8 and later, available both as pure Python and optionally compiled with mypyc for performance.

Low friction: pure Python wheel with no runtime dependencies. Actively maintained with a recent release (195 days ago) and ongoing commits. Optional mypyc compilation available for performance but not required.

MIT License permits unrestricted use, modification, and redistribution with minimal restrictions—suitable for both open-source and commercial projects.

Usage

pip install pytokens
python -m pytokens path/to/file.py

Requires Python 3.8 or later; mypyc compilation is optional and controlled via PYTOKENS_USE_MYPYC environment variable.

Verdict: A lightweight, actively maintained tokenizer with zero dependencies and permissive licensing. No known vulnerabilities and broad Python version support (3.8–3.14) make it a low-risk addition. The optional mypyc compilation path adds performance without mandatory overhead.

Needs verification

Whether the mypyc-compiled variant is available for all supported platforms or only specific architectures.
Performance benchmarks comparing the pure Python and mypyc-compiled versions to understand the practical speed gain.

python tokenizerpython lexerast tokenizationpython code parsingpython 3.14 tokenizerfast python tokenizerspec compliant tokenizer