fastokens

What it is and what it does

fastokens is a Rust-backed tokenizer designed to accelerate BPE encoding for large language models. It loads both HuggingFace tokenizer.json files and tiktoken model files, and ships prebuilt wheels for Python 3.9+ across major platforms, eliminating the need to compile Rust code during installation.

The main use case is speeding up tokenization in inference pipelines where prompt encoding becomes a bottleneck. It includes features like vocabulary extension through add_tokens and add_special_tokens, optional prefix caching for shared system prompts, and PCRE2 resource limits to guard against pathological regex patterns. The package can be used standalone or integrated with serving frameworks.

Use it for:

• Accelerate tokenization in inference pipelines to reduce encoding latency on large prompts.
• Tokenize long contexts faster by enabling the optional prefix cache for repeated system prompts.
• Load and use tiktoken models (e.g., cl100k_base or o200k_base) directly without additional dependencies.
• Extend model vocabularies by adding placeholder tokens to match padded embedding matrices.
• Replace default tokenizers in production serving to improve time-to-first-token metrics.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

fastokens is a high-performance BPE tokenizer for large language models, built on a Rust backend and compatible with HuggingFace tokenizer.json and tiktoken model formats.

Yes, if you need faster tokenization in LLM inference and your model is among the tested families (Qwen, Kimi, Minimax, Gemma, DeepSeek) or you verify compatibility. The prebuilt wheels and zero runtime dependencies make installation frictionless. However, verify the license status in the repository before use in proprietary projects, and confirm that unsupported tokenizer features do not block your use case.

Install

fastokens on PyPI

pip install fastokens

uv add fastokens

poetry add fastokens

Installing fastokens

Before you install

Prebuilt wheels for Python 3.9+ across Linux, macOS, and Windows minimize installation friction. The package is actively maintained with a recent release and no known vulnerabilities.

License in practice

License status is unclear; no SPDX identifier or raw license text is available in the package metadata. Verify the repository's LICENSE file before adopting in proprietary or copyleft-sensitive projects.

Quickstart

pip install fastokens

from fastokens._native import Tokenizer
tokenizer = Tokenizer.from_model("deepseek-ai/DeepSeek-V3.2")
tokens = tokenizer.encode("A very long prompt that is now lightning fast.")

Requires Python 3.9 or later.

Verify before relying

• Whether the 10x+ performance improvement claim applies to your specific workload and model size.
• Exact scope of features not supported compared to the tokenizers library.
• Whether your target model is among the tested families (Qwen, Kimi, Minimax, Gemma, DeepSeek) or requires verification.

Package facts

Evidence: fastokens-0.3.1-cp39-abi3-macosx_10_12_x86_64.whl; fastokens-0.3.1-cp39-abi3-macosx_11_0_arm64.whl; fastokens-0.3.1-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl; fastokens-0.3.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; fastokens-0.3.1-cp39-abi3-manylinux_2_28_aarch64.whl; fastokens-0.3.1-cp39-abi3-manylinux_2_28_armv7l.whl; fastokens-0.3.1-cp39-abi3-manylinux_2_28_ppc64le.whl; fastokens-0.3.1-cp39-abi3-musllinux_1_2_aarch64.whl; fastokens-0.3.1-cp39-abi3-musllinux_1_2_armv7l.whl; fastokens-0.3.1-cp39-abi3-musllinux_1_2_i686.whl; fastokens-0.3.1-cp39-abi3-musllinux_1_2_x86_64.whl; fastokens-0.3.1-cp39-abi3-win_amd64.whl

Tags

More Linguistic packages