tokenizers

tokenizers Permissive license Active ★ 10,964 v0.23.1 released 2026-04-27

Install

tokenizers on PyPI

pip

pip install tokenizers

uv

uv add tokenizers

poetry

poetry add tokenizers

Package facts

License	not declared (permissive)
Python support	supports the current Python release (>=3.10)
Install friction	medium — platform-specific wheel
Runtime dependencies	1 — huggingface-hub
Maintenance	actively maintained — 108 days since the last release
Last repo commit	2026-08-13
First released	2019-11-01
Popularity	one of the top 1,000 most-downloaded packages on PyPI (30-day window, as of 2026-08-13)
Known vulnerabilities	none known (OSV.dev, checked 2026-08-13)

Evidence: tokenizers-0.23.1-cp310-abi3-macosx_10_12_x86_64.whl; tokenizers-0.23.1-cp310-abi3-macosx_11_0_arm64.whl; tokenizers-0.23.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; tokenizers-0.23.1-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl; tokenizers-0.23.1-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl; tokenizers-0.23.1-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl; tokenizers-0.23.1-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl; tokenizers-0.23.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; tokenizers-0.23.1-cp310-abi3-manylinux_2_31_riscv64.whl; tokenizers-0.23.1-cp310-abi3-musllinux_1_2_aarch64.whl; tokenizers-0.23.1-cp310-abi3-musllinux_1_2_armv7l.whl; tokenizers-0.23.1-cp310-abi3-musllinux_1_2_i686.whl; tokenizers-0.23.1-cp310-abi3-musllinux_1_2_x86_64.whl; tokenizers-0.23.1-cp310-abi3-win32.whl; tokenizers-0.23.1-cp310-abi3-win_amd64.whl; tokenizers-0.23.1-cp310-abi3-win_arm64.whl

Keywords: NLP, tokenizer, BPE, transformer, deep learning

Development Status :: 5 - Production/StableIntended Audience :: DevelopersIntended Audience :: EducationIntended Audience :: Science/ResearchLicense :: OSI Approved :: Apache Software LicenseOperating System :: OS IndependentProgramming Language :: Python :: 3Programming Language :: Python :: 3 :: OnlyProgramming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Topic :: Scientific/Engineering :: Artificial Intelligence

About tokenizers

from the package's own PyPI description — quoted content, verbatim

Tokenizers

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.

Bindings over the Rust implementation. If you are interested in the High-level design, you can go check it there.

Otherwise, let's dive in!

Main features:

Train new vocabularies and tokenize using 4 pre-made tokenizers (Bert WordPiece and the 3 most common BPE versions).
Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes less than 20 seconds to tokenize a GB of text on a server's CPU.
Easy to use, but also extremely versatile.
Designed for...

Read as markdown · JSON record · Source repository · Homepage

AI interpretation — verify before relying

AI-generated interpretation of the package facts above; every digit, version, license, or vulnerability id it cites is grounded in the facts already shown on this page

Tokenizers provides fast, production-grade implementations of modern NLP tokenizers (BPE, WordPiece, SentencePiece) with Rust-backed performance, supporting training, encoding, and full preprocessing pipelines.

Medium install friction due to compiled Rust bindings, but extensive prebuilt wheels cover Python 3.10+ across macOS, Linux, Windows, and ARM architectures. Active maintenance with recent releases and strong GitHub presence signal solid reliability.

Licensed under Apache Software License (permissive), allowing commercial and private use with minimal restrictions—standard for open-source ML tooling.

Usage

pip install tokenizers
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained("bert-base-cased")
encoded = tokenizer.encode("Hello world")
print(encoded.ids, encoded.tokens)

Requires Python ≥3.10. Compiled wheels available for most platforms; building from source requires Rust toolchain.

Verdict: Tokenizers 0.23.1 is a mature, actively maintained library with no known vulnerabilities, permissive licensing, and broad platform support via prebuilt wheels. Medium install friction is offset by comprehensive wheel coverage and active development. Suitable for production NLP pipelines.

Needs verification

Whether huggingface-hub dependency adds significant transitive overhead or network requirements at runtime.
Performance benchmarks comparing to alternative tokenization libraries for specific use cases.
Thread-safety guarantees on Python versions below 3.14t beyond what the free-threading section documents.
Actual GitHub star count and community adoption metrics beyond the fact sheet's reference.

NLP tokenization libraryBPE tokenizer implementationfast text tokenizationtransformer tokenizertrain custom tokenizersBERT WordPiece tokenizerbyte-level BPE