---
id: tokenizers
version: "0.23.1"
license: unclear
license_treatment: permissive
maintenance: active
---

# tokenizers

License: permissive · Maintenance: active · Popularity: top 1,000 on PyPI

## Install

pip install tokenizers
uv add tokenizers
poetry add tokenizers

## Description

<p align="center">
    <br>
    <img src="https://huggingface.co/landing/assets/tokenizers/tokenizers-logo.png" width="600"/>
    <br>
<p>
<p align="center">
    <a href="https://badge.fury.io/py/tokenizers">
         <img alt="Build" src="https://badge.fury.io/py/tokenizers.svg">
    </a>
    <a href="https://github.com/huggingface/tokenizers/blob/master/LICENSE">
        <img alt="GitHub" src="https://img.shields.io/github/license/huggingface/tokenizers.svg?color=blue">
    </a>
</p>
<br>

# Tokenizers

Provides an implementation of today's most used tokenizers, with a focus on performance and
versatility.

Bindings over the [Rust](https://github.com/huggingface/tokenizers/tree/master/tokenizers) implementation.
If you are interested in the High-level design, you can go check it there.

Otherwise, let's dive in!

## Main features:

 - Train new vocabularies and tokenize using 4 pre-made tokenizers (Bert WordPiece and the 3
   most common BPE versions).
 - Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes
   less than 20 seconds to tokenize a GB of text on a server's CPU.
 - Easy to use, but also extremely versatile.
 - Designed for...

## AI interpretation — verify before relying

Tokenizers provides fast, production-grade implementations of modern NLP tokenizers (BPE, WordPiece, SentencePiece) with Rust-backed performance, supporting training, encoding, and full preprocessing pipelines.

Verdict: Tokenizers 0.23.1 is a mature, actively maintained library with no known vulnerabilities, permissive licensing, and broad platform support via prebuilt wheels. Medium install friction is offset by comprehensive wheel coverage and active development. Suitable for production NLP pipelines.

[View on SkillFed](https://skillfed.io/packages/tokenizers) · [View on PyPI](https://pypi.org/project/tokenizers/)