{"categories":[{"label":"Scientific/Engineering","url":"https://skillfed.io/packages/category/scientific-engineering/3"}],"enrichment":{"capability":"Provides BPE, Byte BPE, Unigram, and Wordpiece tokenizers for breaking text into subword units, with native implementations and sentencepiece bindings.","skillfed_tags":["nlp","tokenization","transformers"],"use_cases":["Tokenizing text for BERT-like models using the native Wordpiece implementation.","Processing text with BPE or Byte BPE algorithms for GPT-2 or RoBERTa-style models.","Integrating lightweight tokenization into NLP pipelines via the curated-transformers library.","Experimenting with multiple tokenizer types (Unigram, BPE) for model evaluation.","Building custom transformer applications that require subword segmentation."],"what_it_does":"Curated Tokenizers is a lightweight Python library that implements multiple subword tokenization algorithms commonly used in transformer models. It supports BPE (via sentencepiece), Byte BPE (native), Unigram (via sentencepiece), and Wordpiece (native), with the latter two being the tokenizers used in models like XLM-RoBERTa and BERT respectively. The package depends only on the regex library and provides compiled wheels for Python 3.9\u20133.12 across macOS, Linux, and Windows.\n\nThe library is positioned as experimental, with the description explicitly warning that APIs may change incompatibly. The package is dormant (no releases for 851 days as of the fact sheet date), though the repository remains unarchived. The description recommends using curated-transformers as the primary entry point, which provides additional functionality including model loading from Hugging Face Hub.","worth_installing":"Yes, with conditions. The package is permissively licensed (MIT) and has no known vulnerabilities. However, it is explicitly experimental with a dormant maintenance status (851 days since last release). Install only if you need lightweight tokenization for specific transformer models and can tolerate potential API instability, or prefer using it through the curated-transformers library rather than directly."},"id":"curated-tokenizers","links":{"html":"https://skillfed.io/packages/curated-tokenizers","md":"https://skillfed.io/packages/curated-tokenizers.md","pypi":"https://pypi.org/project/curated-tokenizers/"},"maintenance":{"status":"dormant"},"meta":{"latest_release":"2024-04-15","license_spdx":null,"license_treatment":"permissive","name":"curated-tokenizers","python_support":"supports_current","summary":"Lightweight piece tokenization library"},"popularity":{"monthly_downloads":934833,"position":4696,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"2.0.0"}