{"categories":[{"label":"Linguistic","url":"https://skillfed.io/packages/category/text-processing-linguistic"}],"enrichment":{"capability":"fastokens is a high-performance BPE tokenizer for large language models, built on a Rust backend and compatible with HuggingFace tokenizer.json and tiktoken model formats.","skillfed_tags":["llm-inference","tokenization","performance"],"use_cases":["Accelerate tokenization in inference pipelines to reduce encoding latency on large prompts.","Tokenize long contexts faster by enabling the optional prefix cache for repeated system prompts.","Load and use tiktoken models (e.g., cl100k_base or o200k_base) directly without additional dependencies.","Extend model vocabularies by adding placeholder tokens to match padded embedding matrices.","Replace default tokenizers in production serving to improve time-to-first-token metrics."],"what_it_does":"fastokens is a Rust-backed tokenizer designed to accelerate BPE encoding for large language models. It loads both HuggingFace tokenizer.json files and tiktoken model files, and ships prebuilt wheels for Python 3.9+ across major platforms, eliminating the need to compile Rust code during installation.\n\nThe main use case is speeding up tokenization in inference pipelines where prompt encoding becomes a bottleneck. It includes features like vocabulary extension through add_tokens and add_special_tokens, optional prefix caching for shared system prompts, and PCRE2 resource limits to guard against pathological regex patterns. The package can be used standalone or integrated with serving frameworks.","worth_installing":"Yes, if you need faster tokenization in LLM inference and your model is among the tested families (Qwen, Kimi, Minimax, Gemma, DeepSeek) or you verify compatibility. The prebuilt wheels and zero runtime dependencies make installation frictionless. However, verify the license status in the repository before use in proprietary projects, and confirm that unsupported tokenizer features do not block your use case."},"id":"fastokens","links":{"html":"https://skillfed.io/packages/fastokens","md":"https://skillfed.io/packages/fastokens.md","pypi":"https://pypi.org/project/fastokens/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-08-04","license_spdx":null,"license_treatment":"unclear","name":"fastokens","python_support":"supports_current","summary":null},"popularity":{"monthly_downloads":551217,"position":6051,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.3.1"}