blingfire
Python wrapper of lightning fast Finite State Machine based NLP library.
Decision gist · record as of 2026-08-14
Yes, if you need fast tokenization for NLP inference. The package is stable, has no dependencies, and offers significant speed advantages over pure-Python alternatives. However, be aware it is dormant (last release September 2021) and may not receive updates for new Python versions or model formats. Suitable for production use where the existing models and algorithms meet your needs.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Installation is straightforward with no runtime dependencies.
- The package is dormant (last release September 2021, no commits since December 2024), but the underlying C++ library is stable and widely used in production at Microsoft.
License · maintenance · safety
permissive license (permissive) — Licensed under MIT (permissive), allowing free use, modification, and distribution with minimal restrictions.
last release 2021-09-24 (1785 days) · last repo commit 2024-12-08 · 1,893 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 879,860 downloads/mo, #4,823 on PyPI
Alternatives
Verify before relying
pip install blingfire
from blingfire import text_to_words, text_to_sentences
text = 'Hello world. This is a test.'
print(text_to_sentences(text))
print(text_to_words(text))- Whether the package works with Python versions beyond 3.x (exact version support unspecified in metadata)
- Current maintenance status and whether dormancy affects compatibility with modern Python tooling
What it is and what it does
Blingfire is a Python wrapper around Microsoft's finite state machine–based NLP library, designed for high-performance text tokenization and linguistic operations. It provides a unified interface across multiple tokenization algorithms (pattern-based, WordPiece, SentencePiece variants, and BPE) and ships with prebuilt models for popular frameworks like BERT, XLNET, GPT-2, and XLM-RoBERTa, as well as multilingual models for 80+ languages.
The library is optimized for low-latency inference and requires no runtime dependencies beyond Python itself. Models are loaded on demand from binary files, and the package includes default models for sentence breaking and word tokenization that work without additional configuration. It's particularly useful when you need fast, production-grade tokenization that outperforms pure-Python alternatives.
Use it for
- Tokenizing text for BERT, XLNET, or GPT-2 models in inference pipelines where latency matters
- Sentence segmentation and word tokenization in high-throughput NLP applications
- Multilingual text processing using prebuilt models trained on 80+ languages
- Custom tokenization workflows by loading your own finite state machine models
- Replacing slower tokenizers (SpaCy, Hugging Face) when speed is critical
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you need fast tokenization for NLP inference.
The package is stable, has no dependencies, and offers significant speed advantages over pure-Python alternatives. However, be aware it is dormant (last release September 2021) and may not receive updates for new Python versions or model formats. Suitable for production use where the existing models and algorithms meet your needs.
Install
blingfire on PyPI
Before you install
Installation is straightforward with no runtime dependencies. The package is dormant (last release September 2021, no commits since December 2024), but the underlying C++ library is stable and widely used in production at Microsoft.
License in practice
Licensed under MIT (permissive), allowing free use, modification, and distribution with minimal restrictions.
Quickstart
pip install blingfire
from blingfire import text_to_words, text_to_sentences
text = 'Hello world. This is a test.'
print(text_to_sentences(text))
print(text_to_words(text))Verify before relying
- Whether the package works with Python versions beyond 3.x (exact version support unspecified in metadata)
- Current maintenance status and whether dormancy affects compatibility with modern Python tooling
Package facts
| License | permissive license permissive |
| Python support | Not specified |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | None |
| Maintenance | Dormant 1,785 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 879,860 / month, #4,823 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | License :: OSI Approved :: MIT LicenseOperating System :: OS IndependentProgramming Language :: Python :: 3 |
Evidence: blingfire-0.1.8-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “fast tokenization nlp”
- blingfireBlingfire provides fast tokenization and text processing using finite…
- livekit-blingfireProvides BlingFire tokenization bindings for the LiveKit Agents…
- rjiebaA Python binding to the Rust-based jieba-rs Chinese text segmentation…
Give your agent the search over MCP, or paste the wish link into any chat.
More Linguistic packages
Detects and normalizes text encoding from unknown or ambiguous sources, supporting all IANA character sets that Python's core library provides codecs for, with the ability to register custom codecs.
tiktoken is a fast BPE tokenizer that converts text into token sequences compatible with OpenAI models, supporting multiple encoding schemes including o200k_base and model-specific encodings.
Install it if you work with OpenAI APIs or need to understand token boundaries in GPT-family models.
Detects character encoding and language in byte sequences with high accuracy, supporting 99 encodings and returning confidence scores, language tags, and MIME types.
Install it if you need to detect character encoding or language in byte data; the rewrite makes it substantially faster and more accurate than its predecessors.
Converts Unicode text to ASCII by transliterating non-ASCII characters into their closest ASCII equivalents, with no runtime dependencies.
However, if transliteration quality or ongoing maintenance matters, consider unidecode instead despite its GPL-only license.
Lark is a parsing library that builds abstract syntax trees from context-free grammars, supporting multiple parsing algorithms (Earley, LALR(1), CYK) with automatic line and column tracking.
Python bindings to the tree-sitter parsing library, enabling incremental parsing and syntax tree analysis for source code.
See also curated-tokenizers · tokenizers · tokie · torchtext · livekit-blingfire · fastokens · tokenizer · snowballstemmer · tensorflow-text · jieba3k