curated-tokenizers

What it is and what it does

Curated Tokenizers is a lightweight Python library that implements multiple subword tokenization algorithms commonly used in transformer models. It supports BPE (via sentencepiece), Byte BPE (native), Unigram (via sentencepiece), and Wordpiece (native), with the latter two being the tokenizers used in models like XLM-RoBERTa and BERT respectively. The package depends only on the regex library and provides compiled wheels for Python 3.9–3.12 across macOS, Linux, and Windows.

The library is positioned as experimental, with the description explicitly warning that APIs may change incompatibly. The package is dormant (no releases for 851 days as of the fact sheet date), though the repository remains unarchived. The description recommends using curated-transformers as the primary entry point, which provides additional functionality including model loading from Hugging Face Hub.

Use it for:

• Tokenizing text for BERT-like models using the native Wordpiece implementation.
• Processing text with BPE or Byte BPE algorithms for GPT-2 or RoBERTa-style models.
• Integrating lightweight tokenization into NLP pipelines via the curated-transformers library.
• Experimenting with multiple tokenizer types (Unigram, BPE) for model evaluation.
• Building custom transformer applications that require subword segmentation.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides BPE, Byte BPE, Unigram, and Wordpiece tokenizers for breaking text into subword units, with native implementations and sentencepiece bindings.

Yes, with conditions. The package is permissively licensed (MIT) and has no known vulnerabilities. However, it is explicitly experimental with a dormant maintenance status (851 days since last release). Install only if you need lightweight tokenization for specific transformer models and can tolerate potential API instability, or prefer using it through the curated-transformers library rather than directly.

Install

curated-tokenizers on PyPI

pip install curated-tokenizers

uv add curated-tokenizers

poetry add curated-tokenizers

Installing curated-tokenizers

Before you install

Medium install friction due to compiled wheels across multiple Python versions and architectures. Package is dormant (last commit 2024-04-15, no releases in 851 days), though the repository remains active and unarchived.

License in practice

MIT license permits commercial and private use with minimal restrictions; suitable for most projects.

Quickstart

pip install curated_tokenizers

import curated_tokenizers
# Tokenizer instantiation and usage depend on specific tokenizer type (BPE, Wordpiece, etc.)

Requires Python 3.9 or later; compiled wheels available for macOS (Intel and ARM), Linux (x86_64 and aarch64), and Windows (x86_64).

Verify before relying

• Whether the package is actively maintained despite dormancy status and whether breaking API changes are still expected given the 'experimental' warning in the description.
• Specific usage patterns and API stability for each tokenizer type (BPE, Byte BPE, Unigram, Wordpiece).
• Integration requirements with curated-transformers library and whether it is a hard dependency or optional.

Package facts

Evidence: curated_tokenizers-2.0.0-cp310-cp310-macosx_10_9_x86_64.whl; curated_tokenizers-2.0.0-cp310-cp310-macosx_11_0_arm64.whl; curated_tokenizers-2.0.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; curated_tokenizers-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; curated_tokenizers-2.0.0-cp310-cp310-win_amd64.whl; curated_tokenizers-2.0.0-cp311-cp311-macosx_10_9_x86_64.whl; curated_tokenizers-2.0.0-cp311-cp311-macosx_11_0_arm64.whl; curated_tokenizers-2.0.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; curated_tokenizers-2.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; curated_tokenizers-2.0.0-cp311-cp311-win_amd64.whl; curated_tokenizers-2.0.0-cp312-cp312-macosx_10_9_x86_64.whl; curated_tokenizers-2.0.0-cp312-cp312-macosx_11_0_arm64.whl; curated_tokenizers-2.0.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; curated_tokenizers-2.0.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; curated_tokenizers-2.0.0-cp312-cp312-win_amd64.whl; curated_tokenizers-2.0.0-cp39-cp39-macosx_10_9_x86_64.whl; curated_tokenizers-2.0.0-cp39-cp39-macosx_11_0_arm64.whl; curated_tokenizers-2.0.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; curated_tokenizers-2.0.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; curated_tokenizers-2.0.0-cp39-cp39-win_amd64.whl

Tags

More Scientific/Engineering packages