{"categories":[{"label":"Python Modules","url":"https://skillfed.io/packages/category/software-development-libraries-python-modules/16"},{"label":"Artificial Intelligence","url":"https://skillfed.io/packages/category/scientific-engineering-artificial-intelligence/7"}],"enrichment":{"capability":"Converts HuggingFace tokenizers into OpenVINO models for text processing, enabling tokenization and detokenization within OpenVINO inference pipelines without external tokenizer dependencies.","skillfed_tags":["nlp-inference","model-conversion","edge-deployment"],"use_cases":["Deploy end-to-end NLP models as a single OpenVINO artifact without external tokenizer dependencies.","Integrate text preprocessing into OpenVINO inference pipelines for edge or server deployments.","Convert tokenizers to OpenVINO format for use in C++ or Python inference applications.","Add greedy decoding pipelines to text generation models within the OpenVINO framework.","Reduce deployment footprint by eliminating separate tokenizer library installations."],"what_it_does":"OpenVINO Tokenizers bridges tokenizer conversion and OpenVINO's inference runtime by converting tokenizers into compiled OpenVINO models. This lets you perform text tokenization and detokenization as part of an OpenVINO inference graph, eliminating the need to load separate tokenizer libraries during deployment. The package provides both a CLI tool and Python API for converting tokenizers, and supports combining tokenizers with language models into single deployable units.\n\nThe package is built on openvino as its sole runtime dependency and targets modern Python (3.10+). It ships precompiled wheels for macOS (arm64, x86_64), Linux (x86_64, aarch64), and Windows. Tokenization runs on CPU only. Installation can be minimal for using pre-converted tokenizers or include extras for conversion workflows.","worth_installing":"Yes, if you are building OpenVINO-based NLP inference pipelines and want to eliminate external tokenizer dependencies. Active maintenance, permissive Apache-2.0 license, and a recent release (10 days old) support reliability. Install with caution if you require GPU tokenization; CPU-only inference and medium install friction are real trade-offs."},"id":"openvino-tokenizers","links":{"html":"https://skillfed.io/packages/openvino-tokenizers","md":"https://skillfed.io/packages/openvino-tokenizers.md","pypi":"https://pypi.org/project/openvino-tokenizers/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-08-04","license_spdx":null,"license_treatment":"permissive","name":"openvino-tokenizers","python_support":"supports_current","summary":"Convert tokenizers into OpenVINO models"},"popularity":{"monthly_downloads":207975,"position":9538,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"2026.3.0.0"}