{"categories":[{"label":"Software Development","url":"https://skillfed.io/packages/category/software-development/8"},{"label":"Libraries","url":"https://skillfed.io/packages/category/software-development-libraries/5"},{"label":"Scientific/Engineering","url":"https://skillfed.io/packages/category/scientific-engineering/4"},{"label":"Python Modules","url":"https://skillfed.io/packages/category/software-development-libraries-python-modules/12"},{"label":"Artificial Intelligence","url":"https://skillfed.io/packages/category/scientific-engineering-artificial-intelligence/5"},{"label":"Mathematics","url":"https://skillfed.io/packages/category/scientific-engineering-mathematics/2"}],"enrichment":{"capability":"Provides C++ implementations of multiple tokenizers (SentencePiece, Tiktoken, Huggingface, Llama2.c, Tekken) with Python bindings for use in PyTorch-based LLM applications and ExecuTorch runners.","skillfed_tags":["llm-inference","tokenization","executorch"],"use_cases":["Building LLM inference engines with ExecuTorch by selecting and configuring the appropriate tokenizer for model deployment.","Preprocessing text for Mistral-based models using Tekken tokenizer with special token and multilingual support.","Integrating multiple tokenizer backends into a single PyTorch application without managing separate dependencies.","Developing AOT Inductor-based LLM runners that require efficient, compiled tokenization.","Handling multilingual and emoji-rich text in production LLM pipelines with Tekken's Unicode support."],"what_it_does":"pytorch-tokenizers is a Meta-maintained package that bundles C++ implementations of multiple tokenizer algorithms\u2014SentencePiece, Tiktoken, Huggingface-compatible, Llama2.c, and Tekken\u2014with Python bindings. It's designed to support PyTorch-based LLM inference pipelines, particularly within the ExecuTorch and AOT Inductor stacks. The package wraps existing tokenizer implementations (adapted from Google's SentencePiece, Mistral's Tekken, and others) and exposes them as a unified Python interface.\n\nThe package depends on tiktoken, tokenizers, and sentencepiece as runtime dependencies, meaning it acts as a thin wrapper and orchestration layer rather than reimplementing tokenization from scratch. Tekken support includes special token recognition, multilingual Unicode handling, and emoji support. The package is actively maintained, supports Python 3.10 through 3.14, and distributes precompiled wheels for common platforms, reducing build friction for most users.","worth_installing":"Yes, with conditions. Install if you are building LLM inference on ExecuTorch or AOT Inductor and need a unified tokenizer interface with prebuilt wheels. The package is actively maintained, permissively licensed, and has no known vulnerabilities. Avoid if you need only a single tokenizer (use the underlying dependency directly) or if you require Python versions below 3.10. The early release status (Beta, first release 2025-10-20) suggests API stability should be verified for production use."},"id":"pytorch-tokenizers","links":{"html":"https://skillfed.io/packages/pytorch-tokenizers","md":"https://skillfed.io/packages/pytorch-tokenizers.md","pypi":"https://pypi.org/project/pytorch-tokenizers/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-08-05","license_spdx":null,"license_treatment":"permissive","name":"pytorch-tokenizers","python_support":"supports_current","summary":"A package with common tokenizers in Python and C++"},"popularity":{"monthly_downloads":452674,"position":6581,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"1.4.1"}