curated-tokenizers
Lightweight piece tokenization library
Decision gist · record as of 2026-08-14
Yes, with conditions. The package is permissively licensed (MIT) and has no known vulnerabilities. However, it is explicitly experimental with a dormant maintenance status (851 days since last release). Install only if you need lightweight tokenization for specific transformer models and can tolerate potential API instability, or prefer using it through the curated-transformers library rather than directly.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.9 or later; compiled wheels available for macOS (Intel and ARM), Linux (x86_64 and aarch64), and Windows (x86_64).
- Medium install friction due to compiled wheels across multiple Python versions and architectures.
- Package is dormant (last commit 2024-04-15, no releases in 851 days), though the repository remains active and unarchived.
License · maintenance · safety
MIT (permissive) — MIT license permits commercial and private use with minimal restrictions; suitable for most projects.
last release 2024-04-15 (851 days) · last repo commit 2024-04-15 · 12 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 934,833 downloads/mo, #4,696 on PyPI
Alternatives
Verify before relying
pip install curated_tokenizers
import curated_tokenizers
# Tokenizer instantiation and usage depend on specific tokenizer type (BPE, Wordpiece, etc.)- Whether the package is actively maintained despite dormancy status and whether breaking API changes are still expected given the 'experimental' warning in the description.
- Specific usage patterns and API stability for each tokenizer type (BPE, Byte BPE, Unigram, Wordpiece).
- Integration requirements with curated-transformers library and whether it is a hard dependency or optional.
What it is and what it does
Curated Tokenizers is a lightweight Python library that implements multiple subword tokenization algorithms commonly used in transformer models. It supports BPE (via sentencepiece), Byte BPE (native), Unigram (via sentencepiece), and Wordpiece (native), with the latter two being the tokenizers used in models like XLM-RoBERTa and BERT respectively. The package depends only on the regex library and provides compiled wheels for Python 3.9–3.12 across macOS, Linux, and Windows.
The library is positioned as experimental, with the description explicitly warning that APIs may change incompatibly. The package is dormant (no releases for 851 days as of the fact sheet date), though the repository remains unarchived. The description recommends using curated-transformers as the primary entry point, which provides additional functionality including model loading from Hugging Face Hub.
Use it for
- Tokenizing text for BERT-like models using the native Wordpiece implementation.
- Processing text with BPE or Byte BPE algorithms for GPT-2 or RoBERTa-style models.
- Integrating lightweight tokenization into NLP pipelines via the curated-transformers library.
- Experimenting with multiple tokenizer types (Unigram, BPE) for model evaluation.
- Building custom transformer applications that require subword segmentation.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, with conditions.
The package is permissively licensed (MIT) and has no known vulnerabilities. However, it is explicitly experimental with a dormant maintenance status (851 days since last release). Install only if you need lightweight tokenization for specific transformer models and can tolerate potential API instability, or prefer using it through the curated-transformers library rather than directly.
Install
curated-tokenizers on PyPI
Before you install
Medium install friction due to compiled wheels across multiple Python versions and architectures. Package is dormant (last commit 2024-04-15, no releases in 851 days), though the repository remains active and unarchived.
Requires Python 3.9 or later; compiled wheels available for macOS (Intel and ARM), Linux (x86_64 and aarch64), and Windows (x86_64).
License in practice
MIT license permits commercial and private use with minimal restrictions; suitable for most projects.
Quickstart
pip install curated_tokenizers
import curated_tokenizers
# Tokenizer instantiation and usage depend on specific tokenizer type (BPE, Wordpiece, etc.)Verify before relying
- Whether the package is actively maintained despite dormancy status and whether breaking API changes are still expected given the 'experimental' warning in the description.
- Specific usage patterns and API stability for each tokenizer type (BPE, Byte BPE, Unigram, Wordpiece).
- Integration requirements with curated-transformers library and whether it is a hard dependency or optional.
Package facts
| License | MIT permissive |
| Python support | Supports the current Python release >=3.9 |
| Install friction | Medium. Platform-specific wheel |
| Runtime dependencies | 1 packageregex |
| Maintenance | Dormant 851 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 934,833 / month, #4,696 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 5 - Production/StableEnvironment :: ConsoleIntended Audience :: DevelopersIntended Audience :: Science/ResearchLicense :: OSI Approved :: MIT LicenseOperating System :: MacOS :: MacOS XProgramming Language :: CythonProgramming Language :: Python :: 3Programming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.9Topic :: Scientific/Engineering |
Evidence: curated_tokenizers-2.0.0-cp310-cp310-macosx_10_9_x86_64.whl; curated_tokenizers-2.0.0-cp310-cp310-macosx_11_0_arm64.whl; curated_tokenizers-2.0.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; curated_tokenizers-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; curated_tokenizers-2.0.0-cp310-cp310-win_amd64.whl; curated_tokenizers-2.0.0-cp311-cp311-macosx_10_9_x86_64.whl; curated_tokenizers-2.0.0-cp311-cp311-macosx_11_0_arm64.whl; curated_tokenizers-2.0.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; curated_tokenizers-2.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; curated_tokenizers-2.0.0-cp311-cp311-win_amd64.whl; curated_tokenizers-2.0.0-cp312-cp312-macosx_10_9_x86_64.whl; curated_tokenizers-2.0.0-cp312-cp312-macosx_11_0_arm64.whl; curated_tokenizers-2.0.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; curated_tokenizers-2.0.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; curated_tokenizers-2.0.0-cp312-cp312-win_amd64.whl; curated_tokenizers-2.0.0-cp39-cp39-macosx_10_9_x86_64.whl; curated_tokenizers-2.0.0-cp39-cp39-macosx_11_0_arm64.whl; curated_tokenizers-2.0.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; curated_tokenizers-2.0.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; curated_tokenizers-2.0.0-cp39-cp39-win_amd64.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “BPE wordpiece unigram”
- curated-tokenizersProvides BPE, Byte BPE, Unigram, and Wordpiece tokenizers for…
- tokieA fast, Rust-backed tokenizer library that encodes and decodes text…
- blingfireBlingfire provides fast tokenization and text processing using finite…
Give your agent the search over MCP, or paste the wish link into any chat.
More Scientific/Engineering packages
NumPy provides an N-dimensional array object and a comprehensive suite of mathematical, linear algebra, Fourier transform, and random number functions for scientific computing in Python.
pandas provides fast, flexible data structures (Series and DataFrame) for loading, cleaning, transforming, and analyzing labeled or relational data in Python.
scipy provides numerical algorithms for mathematics, science, and engineering—including optimization, integration, linear algebra, Fourier transforms, signal and image processing, and ODE solvers—built on numpy arrays.
scikit-learn provides a comprehensive Python library for supervised and unsupervised machine learning, including classification, regression, clustering, dimensionality reduction, and model evaluation tools built on NumPy and SciPy.
Install it if you need to train, evaluate, or deploy supervised or unsupervised learning models.
dill extends Python's pickle module to serialize and deserialize a much wider range of Python objects, including functions, lambdas, classes, and interpreter sessions, to byte streams for storage or network transmission.
Multiprocess is an enhanced fork of Python's standard multiprocessing library that uses dill for better serialization, allowing you to spawn processes with a threading-like API and share complex objects between them.
Install it if you use multiprocessing and encounter pickle serialization limits with lambdas or complex objects.
See also blingfire · bpemb · sentencepiece · tokenizers · tokie · pytorch-tokenizers · spacy-curated-transformers · wordsegment · tiktoken · curated-transformers