pytorch-tokenizers
A package with common tokenizers in Python and C++
Decision gist · record as of 2026-08-14
Yes, with conditions. Install if you are building LLM inference on ExecuTorch or AOT Inductor and need a unified tokenizer interface with prebuilt wheels. The package is actively maintained, permissively licensed, and has no known vulnerabilities. Avoid if you need only a single tokenizer (use the underlying dependency directly) or if you require Python versions below 3.10. The early release status (Beta, first release 2025-10-20) suggests API stability should be verified for production use.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.10 or later; C++ build dependencies needed if installing from source (git clone and submodule setup required).
- Medium install friction due to compiled C++ components; prebuilt wheels available for Python 3.10–3.14 across macOS ARM64, Linux x86_64/aarch64, and Windows.
- Recently active (last commit 2026-08-05, release 9 days old) with maintained repository status.
License · maintenance · safety
permissive license (permissive) — BSD 3-Clause permissive license allows commercial and private use with attribution; additional code covered by MIT and Apache licenses. No restrictions on derivative works or redistribution.
last release 2026-08-05 (9 days) · last repo commit 2026-08-05 · 50 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 452,674 downloads/mo, #6,581 on PyPI
Alternatives
Verify before relying
pip install pytorch-tokenizers
from pytorch_tokenizers import SentencePieceTokenizer
tokenizer = SentencePieceTokenizer(model_path="model.model")
tokens = tokenizer.encode("Hello world")- Performance characteristics and throughput compared to alternative tokenizer implementations.
- Specific version compatibility with ExecuTorch and torchchat ecosystems.
- Tekken tokenizer's claimed 100% decode accuracy and test coverage scope.
- Whether all tokenizer implementations (SentencePiece, Tiktoken, Huggingface, Llama2.c, Tekken) are production-ready or some remain experimental.
What it is and what it does
pytorch-tokenizers is a Meta-maintained package that bundles C++ implementations of multiple tokenizer algorithms—SentencePiece, Tiktoken, Huggingface-compatible, Llama2.c, and Tekken—with Python bindings. It's designed to support PyTorch-based LLM inference pipelines, particularly within the ExecuTorch and AOT Inductor stacks. The package wraps existing tokenizer implementations (adapted from Google's SentencePiece, Mistral's Tekken, and others) and exposes them as a unified Python interface.
The package depends on tiktoken, tokenizers, and sentencepiece as runtime dependencies, meaning it acts as a thin wrapper and orchestration layer rather than reimplementing tokenization from scratch. Tekken support includes special token recognition, multilingual Unicode handling, and emoji support. The package is actively maintained, supports Python 3.10 through 3.14, and distributes precompiled wheels for common platforms, reducing build friction for most users.
Use it for
- Building LLM inference engines with ExecuTorch by selecting and configuring the appropriate tokenizer for model deployment.
- Preprocessing text for Mistral-based models using Tekken tokenizer with special token and multilingual support.
- Integrating multiple tokenizer backends into a single PyTorch application without managing separate dependencies.
- Developing AOT Inductor-based LLM runners that require efficient, compiled tokenization.
- Handling multilingual and emoji-rich text in production LLM pipelines with Tekken's Unicode support.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, with conditions.
Install if you are building LLM inference on ExecuTorch or AOT Inductor and need a unified tokenizer interface with prebuilt wheels. The package is actively maintained, permissively licensed, and has no known vulnerabilities. Avoid if you need only a single tokenizer (use the underlying dependency directly) or if you require Python versions below 3.10. The early release status (Beta, first release 2025-10-20) suggests API stability should be verified for production use.
Install
pytorch-tokenizers on PyPI
Before you install
Medium install friction due to compiled C++ components; prebuilt wheels available for Python 3.10–3.14 across macOS ARM64, Linux x86_64/aarch64, and Windows. Recently active (last commit 2026-08-05, release 9 days old) with maintained repository status.
Requires Python 3.10 or later; C++ build dependencies needed if installing from source (git clone and submodule setup required).
License in practice
BSD 3-Clause permissive license allows commercial and private use with attribution; additional code covered by MIT and Apache licenses. No restrictions on derivative works or redistribution.
Quickstart
pip install pytorch-tokenizers
from pytorch_tokenizers import SentencePieceTokenizer
tokenizer = SentencePieceTokenizer(model_path="model.model")
tokens = tokenizer.encode("Hello world")Verify before relying
- Performance characteristics and throughput compared to alternative tokenizer implementations.
- Specific version compatibility with ExecuTorch and torchchat ecosystems.
- Tekken tokenizer's claimed 100% decode accuracy and test coverage scope.
- Whether all tokenizer implementations (SentencePiece, Tiktoken, Huggingface, Llama2.c, Tekken) are production-ready or some remain experimental.
Package facts
| License | permissive license permissive |
| Python support | Supports the current Python release >=3.10 |
| Install friction | Medium. Platform-specific wheel |
| Runtime dependencies | 3 packagestiktokentokenizerssentencepiece |
| Maintenance | Actively maintained 9 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 452,674 / month, #6,581 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 4 - BetaIntended Audience :: DevelopersIntended Audience :: EducationIntended Audience :: Science/ResearchLicense :: OSI Approved :: BSD LicenseProgramming Language :: C++Programming Language :: Python :: 3Programming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Topic :: Scientific/EngineeringTopic :: Scientific/Engineering :: Artificial IntelligenceTopic :: Scientific/Engineering :: MathematicsTopic :: Software DevelopmentTopic :: Software Development :: LibrariesTopic :: Software Development :: Libraries :: Python Modules |
Evidence: pytorch_tokenizers-1.4.1-cp310-cp310-macosx_14_0_arm64.whl; pytorch_tokenizers-1.4.1-cp310-cp310-manylinux_2_28_aarch64.whl; pytorch_tokenizers-1.4.1-cp310-cp310-manylinux_2_28_x86_64.whl; pytorch_tokenizers-1.4.1-cp310-cp310-win_amd64.whl; pytorch_tokenizers-1.4.1-cp311-cp311-macosx_14_0_arm64.whl; pytorch_tokenizers-1.4.1-cp311-cp311-manylinux_2_28_aarch64.whl; pytorch_tokenizers-1.4.1-cp311-cp311-manylinux_2_28_x86_64.whl; pytorch_tokenizers-1.4.1-cp311-cp311-win_amd64.whl; pytorch_tokenizers-1.4.1-cp312-cp312-macosx_14_0_arm64.whl; pytorch_tokenizers-1.4.1-cp312-cp312-manylinux_2_28_aarch64.whl; pytorch_tokenizers-1.4.1-cp312-cp312-manylinux_2_28_x86_64.whl; pytorch_tokenizers-1.4.1-cp312-cp312-win_amd64.whl; pytorch_tokenizers-1.4.1-cp313-cp313-macosx_14_0_arm64.whl; pytorch_tokenizers-1.4.1-cp313-cp313-manylinux_2_28_aarch64.whl; pytorch_tokenizers-1.4.1-cp313-cp313-manylinux_2_28_x86_64.whl; pytorch_tokenizers-1.4.1-cp313-cp313-win_amd64.whl; pytorch_tokenizers-1.4.1-cp314-cp314-macosx_14_0_arm64.whl; pytorch_tokenizers-1.4.1-cp314-cp314-manylinux_2_28_aarch64.whl; pytorch_tokenizers-1.4.1-cp314-cp314-manylinux_2_28_x86_64.whl; pytorch_tokenizers-1.4.1-cp314-cp314-win_amd64.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “tokenizer for llm inference”
- pytorch-tokenizersProvides C++ implementations of multiple tokenizers (SentencePiece,…
- fastokensfastokens is a high-performance BPE tokenizer for large language…
- tokieA fast, Rust-backed tokenizer library that encodes and decodes text…
Give your agent the search over MCP, or paste the wish link into any chat.
More Software Development packages
Provides backported and experimental type hints for Python 3.9+, allowing use of newer typing features on older Python versions and enabling early experimentation with type system PEPs before they enter the standard library.
NumPy provides an N-dimensional array object and a comprehensive suite of mathematical, linear algebra, Fourier transform, and random number functions for scientific computing in Python.
FastAPI is a Python web framework for building REST APIs using type hints, with automatic request validation, serialization, and interactive API documentation.
Provides a way to document function parameters, class attributes, return types, and variables inline using Python's `Annotated` type hint syntax instead of traditional docstrings.
Typer builds command-line applications from Python functions using type hints, automatically generating help text, argument parsing, and shell completion.
Install it if you are building CLIs in Python.
Distlib provides low-level packaging utilities for building, distributing, and managing Python software—including metadata handling, version specifiers, wheel support, script installation, and dependency resolution.
See also mistral_common · torchtext · tokenizers · tokie · curated-tokenizers · fastokens · tiktoken · pytorch-pretrained-bert · llmcompressor · torchtune