pytorch-tokenizers

What it is and what it does

pytorch-tokenizers is a Meta-maintained package that bundles C++ implementations of multiple tokenizer algorithms—SentencePiece, Tiktoken, Huggingface-compatible, Llama2.c, and Tekken—with Python bindings. It's designed to support PyTorch-based LLM inference pipelines, particularly within the ExecuTorch and AOT Inductor stacks. The package wraps existing tokenizer implementations (adapted from Google's SentencePiece, Mistral's Tekken, and others) and exposes them as a unified Python interface.

The package depends on tiktoken, tokenizers, and sentencepiece as runtime dependencies, meaning it acts as a thin wrapper and orchestration layer rather than reimplementing tokenization from scratch. Tekken support includes special token recognition, multilingual Unicode handling, and emoji support. The package is actively maintained, supports Python 3.10 through 3.14, and distributes precompiled wheels for common platforms, reducing build friction for most users.

Use it for:

• Building LLM inference engines with ExecuTorch by selecting and configuring the appropriate tokenizer for model deployment.
• Preprocessing text for Mistral-based models using Tekken tokenizer with special token and multilingual support.
• Integrating multiple tokenizer backends into a single PyTorch application without managing separate dependencies.
• Developing AOT Inductor-based LLM runners that require efficient, compiled tokenization.
• Handling multilingual and emoji-rich text in production LLM pipelines with Tekken's Unicode support.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides C++ implementations of multiple tokenizers (SentencePiece, Tiktoken, Huggingface, Llama2.c, Tekken) with Python bindings for use in PyTorch-based LLM applications and ExecuTorch runners.

Yes, with conditions. Install if you are building LLM inference on ExecuTorch or AOT Inductor and need a unified tokenizer interface with prebuilt wheels. The package is actively maintained, permissively licensed, and has no known vulnerabilities. Avoid if you need only a single tokenizer (use the underlying dependency directly) or if you require Python versions below 3.10. The early release status (Beta, first release 2025-10-20) suggests API stability should be verified for production use.

Install

pytorch-tokenizers on PyPI

pip install pytorch-tokenizers

uv add pytorch-tokenizers

poetry add pytorch-tokenizers

Installing pytorch-tokenizers

Before you install

Medium install friction due to compiled C++ components; prebuilt wheels available for Python 3.10–3.14 across macOS ARM64, Linux x86_64/aarch64, and Windows. Recently active (last commit 2026-08-05, release 9 days old) with maintained repository status.

License in practice

BSD 3-Clause permissive license allows commercial and private use with attribution; additional code covered by MIT and Apache licenses. No restrictions on derivative works or redistribution.

Quickstart

pip install pytorch-tokenizers

from pytorch_tokenizers import SentencePieceTokenizer
tokenizer = SentencePieceTokenizer(model_path="model.model")
tokens = tokenizer.encode("Hello world")

Requires Python 3.10 or later; C++ build dependencies needed if installing from source (git clone and submodule setup required).

Verify before relying

• Performance characteristics and throughput compared to alternative tokenizer implementations.
• Specific version compatibility with ExecuTorch and torchchat ecosystems.
• Tekken tokenizer's claimed 100% decode accuracy and test coverage scope.
• Whether all tokenizer implementations (SentencePiece, Tiktoken, Huggingface, Llama2.c, Tekken) are production-ready or some remain experimental.

Package facts

Evidence: pytorch_tokenizers-1.4.1-cp310-cp310-macosx_14_0_arm64.whl; pytorch_tokenizers-1.4.1-cp310-cp310-manylinux_2_28_aarch64.whl; pytorch_tokenizers-1.4.1-cp310-cp310-manylinux_2_28_x86_64.whl; pytorch_tokenizers-1.4.1-cp310-cp310-win_amd64.whl; pytorch_tokenizers-1.4.1-cp311-cp311-macosx_14_0_arm64.whl; pytorch_tokenizers-1.4.1-cp311-cp311-manylinux_2_28_aarch64.whl; pytorch_tokenizers-1.4.1-cp311-cp311-manylinux_2_28_x86_64.whl; pytorch_tokenizers-1.4.1-cp311-cp311-win_amd64.whl; pytorch_tokenizers-1.4.1-cp312-cp312-macosx_14_0_arm64.whl; pytorch_tokenizers-1.4.1-cp312-cp312-manylinux_2_28_aarch64.whl; pytorch_tokenizers-1.4.1-cp312-cp312-manylinux_2_28_x86_64.whl; pytorch_tokenizers-1.4.1-cp312-cp312-win_amd64.whl; pytorch_tokenizers-1.4.1-cp313-cp313-macosx_14_0_arm64.whl; pytorch_tokenizers-1.4.1-cp313-cp313-manylinux_2_28_aarch64.whl; pytorch_tokenizers-1.4.1-cp313-cp313-manylinux_2_28_x86_64.whl; pytorch_tokenizers-1.4.1-cp313-cp313-win_amd64.whl; pytorch_tokenizers-1.4.1-cp314-cp314-macosx_14_0_arm64.whl; pytorch_tokenizers-1.4.1-cp314-cp314-manylinux_2_28_aarch64.whl; pytorch_tokenizers-1.4.1-cp314-cp314-manylinux_2_28_x86_64.whl; pytorch_tokenizers-1.4.1-cp314-cp314-win_amd64.whl

Keywords: pytorch, machine learning, llm

Tags

More Software Development packages