---
id: pytokens
version: "0.4.1"
license: MIT License Copyright (c) 2024 Tushar Sadhwani Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to… (full text in the JSON record)
license_treatment: permissive
maintenance: active
---

# pytokens — A Fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.

License: permissive · Maintenance: active · Popularity: top 1,000 on PyPI

## Install

pip install pytokens
uv add pytokens
poetry add pytokens

## Description

# pytokens

A Fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.

## Installation

```bash
pip install pytokens
```

## Usage

```bash
python -m pytokens path/to/file.py
```

## Local Development / Testing

- Create and activate a virtual environment
- Run `PYTOKENS_USE_MYPYC=0 pip install -e '.[dev]'` to do an editable install
- Run `pytest` to run tests

## Type Checking

Run `mypy .`

## Compilation with mypyc

By default, we compile with mypyc.
Use the `PYTOKENS_USE_MYPYC` environment variable to control this.

To check if you are using a compiled version, see whether the output of this is `.py` or `.so`:
```bash
python -c "import pytokens; print(pytokens.__file__)"
```

## Create and upload a package to PyPI

1. Make sure to bump the version in `pyproject.toml`.
2. Push to Github so CI can build the wheels and sdist.
3. Download the artifacts from the CI run.
  - Find the "Build wheels" job, click "Summary", scroll down to the bottom to see "Artifacts"
    and download the `cibw-wheels` artifact.
4. Unzip the artifact.
5. Upload the contents of the artifact to PyPI
  - e.g. via `twine upload cibw-wheels/*`

## AI interpretation — verify before relying

A fast, spec-compliant Python tokenizer that generates tokens for Python 3.14+ code while running on Python 3.8 and later, available both as pure Python and optionally compiled with mypyc for performance.

Verdict: A lightweight, actively maintained tokenizer with zero dependencies and permissive licensing. No known vulnerabilities and broad Python version support (3.8–3.14) make it a low-risk addition. The optional mypyc compilation path adds performance without mandatory overhead.

[View on SkillFed](https://skillfed.io/packages/pytokens) · [View on PyPI](https://pypi.org/project/pytokens/)