openvino-tokenizers
Convert tokenizers into OpenVINO models
Decision gist · record as of 2026-08-14
Yes, if you are building OpenVINO-based NLP inference pipelines and want to eliminate external tokenizer dependencies. Active maintenance, permissive Apache-2.0 license, and a recent release (10 days old) support reliability. Install with caution if you require GPU tokenization; CPU-only inference and medium install friction are real trade-offs.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires openvino runtime dependency; inference runs on CPU device only.
- Medium install friction due to platform-specific wheels (macOS arm64, x86_64, Linux aarch64, Windows) and a required openvino runtime dependency.
- Active maintenance with a release 10 days old suggests ongoing support.
License · maintenance · safety
Apache-2.0 (permissive) — Apache-2.0 permissive license allows commercial and private use with minimal restrictions; suitable for most projects.
last release 2026-08-04 (10 days)
0 known vulnerabilities (OSV.dev, 2026-08-14) · 207,975 downloads/mo, #9,538 on PyPI
Alternatives
Verify before relying
pip install openvino-tokenizers
from openvino_tokenizers import convert_tokenizer
from openvino import compile_model
ov_tokenizer = convert_tokenizer(hf_tokenizer)
compiled = compile_model(ov_tokenizer)- Whether converted tokenizers maintain feature parity with original tokenizers across all model types.
- Performance characteristics (latency, throughput) of OpenVINO tokenizers vs. native implementations.
- Compatibility matrix between openvino-tokenizers and openvino versions beyond the first-three-digit matching guidance.
- Which tokenizer conversion extras (sentencepiece, tiktoken) are required for specific model families.
What it is and what it does
OpenVINO Tokenizers bridges tokenizer conversion and OpenVINO's inference runtime by converting tokenizers into compiled OpenVINO models. This lets you perform text tokenization and detokenization as part of an OpenVINO inference graph, eliminating the need to load separate tokenizer libraries during deployment. The package provides both a CLI tool and Python API for converting tokenizers, and supports combining tokenizers with language models into single deployable units.
The package is built on openvino as its sole runtime dependency and targets modern Python (3.10+). It ships precompiled wheels for macOS (arm64, x86_64), Linux (x86_64, aarch64), and Windows. Tokenization runs on CPU only. Installation can be minimal for using pre-converted tokenizers or include extras for conversion workflows.
Use it for
- Deploy end-to-end NLP models as a single OpenVINO artifact without external tokenizer dependencies.
- Integrate text preprocessing into OpenVINO inference pipelines for edge or server deployments.
- Convert tokenizers to OpenVINO format for use in C++ or Python inference applications.
- Add greedy decoding pipelines to text generation models within the OpenVINO framework.
- Reduce deployment footprint by eliminating separate tokenizer library installations.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you are building OpenVINO-based NLP inference pipelines and want to eliminate external tokenizer dependencies.
Active maintenance, permissive Apache-2.0 license, and a recent release (10 days old) support reliability. Install with caution if you require GPU tokenization; CPU-only inference and medium install friction are real trade-offs.
Install
openvino-tokenizers on PyPI
Before you install
Medium install friction due to platform-specific wheels (macOS arm64, x86_64, Linux aarch64, Windows) and a required openvino runtime dependency. Active maintenance with a release 10 days old suggests ongoing support.
Requires openvino runtime dependency; inference runs on CPU device only.
License in practice
Apache-2.0 permissive license allows commercial and private use with minimal restrictions; suitable for most projects.
Quickstart
pip install openvino-tokenizers
from openvino_tokenizers import convert_tokenizer
from openvino import compile_model
ov_tokenizer = convert_tokenizer(hf_tokenizer)
compiled = compile_model(ov_tokenizer)Verify before relying
- Whether converted tokenizers maintain feature parity with original tokenizers across all model types.
- Performance characteristics (latency, throughput) of OpenVINO tokenizers vs. native implementations.
- Compatibility matrix between openvino-tokenizers and openvino versions beyond the first-three-digit matching guidance.
- Which tokenizer conversion extras (sentencepiece, tiktoken) are required for specific model families.
Package facts
| License | Apache-2.0 permissive |
| Python support | Supports the current Python release >=3.10 |
| Install friction | Medium. Platform-specific wheel |
| Runtime dependencies | 1 packageopenvino |
| Maintenance | Actively maintained 10 days since the last release |
| First released | |
| Downloads | 207,975 / month, #9,538 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 5 - Production/StableIntended Audience :: DevelopersIntended Audience :: Science/ResearchLicense :: OSI Approved :: Apache Software LicenseOperating System :: MacOSOperating System :: Microsoft :: WindowsOperating System :: POSIX :: LinuxOperating System :: UnixProgramming Language :: CProgramming Language :: C++Programming Language :: Python :: 3 :: OnlyProgramming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Topic :: Scientific/Engineering :: Artificial IntelligenceTopic :: Software Development :: Libraries :: Python Modules |
Evidence: openvino_tokenizers-2026.3.0.0-py3-none-macosx_11_0_arm64.whl; openvino_tokenizers-2026.3.0.0-py3-none-manylinux_2_28_x86_64.whl; openvino_tokenizers-2026.3.0.0-py3-none-manylinux_2_31_aarch64.whl; openvino_tokenizers-2026.3.0.0-py3-none-win_amd64.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “tokenizer conversion openvino”
- openvino-tokenizersConverts HuggingFace tokenizers into OpenVINO models for text…
- openvino-devProvides command-line tools and Python APIs to convert deep learning…
- openvino-telemetrySends telemetry data from OpenVINO toolkit components to Google…
Give your agent the search over MCP, or paste the wish link into any chat.
More Python Modules packages
Converts domain names between Unicode and ASCII-compatible encoding (Punycode) according to IDNA 2008 and Unicode Technical Standard 46, with security validation and broader script coverage than the standard library.
Install it if you work with internationalized domain names, need to validate domains, or use HTTP clients that depend on it transitively.
Setuptools is a Python build backend and package management tool that handles building, distributing, and installing Python packages, including support for C/C++ extension modules.
PyYAML parses and emits YAML 1.1 data format, enabling serialization and deserialization of configuration files and Python objects to and from human-readable YAML text.
Pydantic validates Python data structures against type hints, coercing and checking input at runtime to ensure it matches a declared schema.
Provides reusable metadata objects for use with PEP-593 `typing.Annotated` to express common constraints like bounds, collection sizes, and predicates on types.
Install it if you use or build libraries that need to express type constraints in a standardized, inspectable way—or if you want to annotate your own types with…
Provides runtime tools to inspect and introspect Python type annotations, enabling programmatic examination of type hints at execution time.
See also openvino · openvino-genai · optimum-intel · tokie · openvino-dev · tokenizers · pytorch-tokenizers · tensorflow-text · onnxruntime-openvino · nncf