tokenizers
Decision gist · record as of 2026-08-14
Yes. Tokenizers is production-stable, actively maintained, widely used (186th on PyPI by downloads), and has no known vulnerabilities. Install friction is moderate but manageable via pre-built wheels. Use it if you need fast, flexible tokenization for transformer models or want to train custom vocabularies; the Rust backend and alignment tracking justify the compiled dependency.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python >=3.10.
- Building from source requires Rust toolchain; pre-built wheels available for most platforms.
- Medium install friction due to compiled Rust bindings, but pre-built wheels cover most platforms (x86_64, ARM, RISC-V, Windows).
License · maintenance · safety
permissive license (permissive) — Permissive license (Apache) means you can use, modify, and distribute this package freely in commercial and private projects without restriction.
last release 2026-04-27 (109 days) · last repo commit 2026-08-13 · 10,965 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 222,890,981 downloads/mo, #186 on PyPI
Alternatives
Verify before relying
pip install tokenizers
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained("bert-base-cased")
encoded = tokenizer.encode("Hello world")
print(encoded.ids)- Whether huggingface-hub dependency is required at runtime or only for loading pretrained models from Hub
- Performance benchmarks for the 'less than 20 seconds to tokenize a GB' claim on typical hardware
What it is and what it does
Tokenizers is a Python library that breaks text into tokens (words, subwords, or characters) for use in machine learning models, particularly transformers. It wraps a high-performance Rust implementation and provides both pre-built tokenizers (BPE variants and WordPiece) and a composable API to build custom ones. The library handles the full pipeline: normalization, pre-tokenization, model-specific encoding, post-processing, and padding/truncation.
You can load pretrained tokenizers from Hugging Face Hub, train new vocabularies on your own text, or assemble tokenizers from individual components (models, pre-tokenizers, decoders, processors). It tracks alignment between tokens and original text, making it possible to map predictions back to source spans. The Rust backend makes both training and inference fast; the library also supports free-threaded Python (3.14t) with thread-safe mutation via internal locks.
Use it for
- Load a pretrained BERT or GPT tokenizer to preprocess text before feeding it to a transformer model
- Train a custom BPE vocabulary on domain-specific text (medical, legal, code) for specialized models
- Build a multi-stage tokenization pipeline combining byte-level encoding, normalization, and special token insertion
- Tokenize large text corpora (gigabytes) quickly for batch preprocessing in production NLP pipelines
- Recover the original text span for each token to align model predictions with source documents
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
Tokenizers is production-stable, actively maintained, widely used (186th on PyPI by downloads), and has no known vulnerabilities. Install friction is moderate but manageable via pre-built wheels. Use it if you need fast, flexible tokenization for transformer models or want to train custom vocabularies; the Rust backend and alignment tracking justify the compiled dependency.
Install
tokenizers on PyPI
Before you install
Medium install friction due to compiled Rust bindings, but pre-built wheels cover most platforms (x86_64, ARM, RISC-V, Windows). Active maintenance with recent releases; last commit 2026-08-13. Requires Python >=3.10.
Requires Python >=3.10. Building from source requires Rust toolchain; pre-built wheels available for most platforms.
License in practice
Permissive license (Apache) means you can use, modify, and distribute this package freely in commercial and private projects without restriction.
Quickstart
pip install tokenizers
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained("bert-base-cased")
encoded = tokenizer.encode("Hello world")
print(encoded.ids)Verify before relying
- Whether huggingface-hub dependency is required at runtime or only for loading pretrained models from Hub
- Performance benchmarks for the 'less than 20 seconds to tokenize a GB' claim on typical hardware
Package facts
| License | permissive license permissive |
| Python support | Supports the current Python release >=3.10 |
| Install friction | Medium. Platform-specific wheel |
| Runtime dependencies | 1 packagehuggingface-hub |
| Maintenance | Actively maintained 109 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 222,890,981 / month, #186 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 5 - Production/StableIntended Audience :: DevelopersIntended Audience :: EducationIntended Audience :: Science/ResearchLicense :: OSI Approved :: Apache Software LicenseOperating System :: OS IndependentProgramming Language :: Python :: 3Programming Language :: Python :: 3 :: OnlyProgramming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Topic :: Scientific/Engineering :: Artificial Intelligence |
Evidence: tokenizers-0.23.1-cp310-abi3-macosx_10_12_x86_64.whl; tokenizers-0.23.1-cp310-abi3-macosx_11_0_arm64.whl; tokenizers-0.23.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; tokenizers-0.23.1-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl; tokenizers-0.23.1-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl; tokenizers-0.23.1-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl; tokenizers-0.23.1-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl; tokenizers-0.23.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; tokenizers-0.23.1-cp310-abi3-manylinux_2_31_riscv64.whl; tokenizers-0.23.1-cp310-abi3-musllinux_1_2_aarch64.whl; tokenizers-0.23.1-cp310-abi3-musllinux_1_2_armv7l.whl; tokenizers-0.23.1-cp310-abi3-musllinux_1_2_i686.whl; tokenizers-0.23.1-cp310-abi3-musllinux_1_2_x86_64.whl; tokenizers-0.23.1-cp310-abi3-win32.whl; tokenizers-0.23.1-cp310-abi3-win_amd64.whl; tokenizers-0.23.1-cp310-abi3-win_arm64.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “BPE tokenizer”
- tokenizersTokenizers converts raw text into token sequences for NLP models,…
- tokieA fast, Rust-backed tokenizer library that encodes and decodes text…
- tiktokentiktoken is a fast BPE tokenizer that converts text into token…
Give your agent the search over MCP, or paste the wish link into any chat.
More Artificial Intelligence packages
LiteLLM provides a unified Python interface to call 100+ LLM providers (OpenAI, Anthropic, Gemini, Bedrock, Azure, and others) using OpenAI-compatible API format, available as both a Python SDK and a self-hosted AI Gateway proxy server.
Install it if you need to work with multiple LLM providers or want to centralize LLM routing in your organization.
Client library and CLI tool for downloading, uploading, and managing models, datasets, and repositories on the Hugging Face Hub platform.
Install it if you work with Hugging Face Hub models or datasets.
LangChain provides a framework for building agents and LLM-powered applications by composing language models, tools, and memory through a unified API that abstracts over multiple model providers.
hf-xet provides chunk-based deduplication and efficient file transfer for the Hugging Face Hub, enabling faster uploads and downloads of large files with local disk caching.
Transformers provides a unified framework for loading, fine-tuning, and running state-of-the-art pretrained models across text, vision, audio, video, and multimodal tasks using PyTorch, JAX, or TensorFlow.
Install it if you need to run or train any transformer-based model for NLP, vision, audio, or multimodal tasks.
LangChain Core provides the foundational abstractions and interfaces for building LLM applications, enabling modular composition of language model chains, agents, and tools across the LangChain ecosystem.
Install it if you are building any LLM application that benefits from modular abstractions and ecosystem interoperability.
See also curated-tokenizers · tokie · pytorch-tokenizers · blingfire · fastokens · tiktoken · pytokens · spacy-alignments · openvino-tokenizers · tensorflow-text