---
id: blingfire
version: "0.1.8"
license: unclear
license_treatment: permissive
maintenance: dormant
---

# blingfire — Python wrapper of lightning fast Finite State Machine based NLP library.

License: permissive · Maintenance: dormant · Downloads: 879.9K/mo

## What it is and what it does

Blingfire is a Python wrapper around Microsoft's finite state machine–based NLP library, designed for high-performance text tokenization and linguistic operations. It provides a unified interface across multiple tokenization algorithms (pattern-based, WordPiece, SentencePiece variants, and BPE) and ships with prebuilt models for popular frameworks like BERT, XLNET, GPT-2, and XLM-RoBERTa, as well as multilingual models for 80+ languages.

The library is optimized for low-latency inference and requires no runtime dependencies beyond Python itself. Models are loaded on demand from binary files, and the package includes default models for sentence breaking and word tokenization that work without additional configuration. It's particularly useful when you need fast, production-grade tokenization that outperforms pure-Python alternatives.

Use it for:

- Tokenizing text for BERT, XLNET, or GPT-2 models in inference pipelines where latency matters
- Sentence segmentation and word tokenization in high-throughput NLP applications
- Multilingual text processing using prebuilt models trained on 80+ languages
- Custom tokenization workflows by loading your own finite state machine models
- Replacing slower tokenizers (SpaCy, Hugging Face) when speed is critical

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Blingfire provides fast tokenization and text processing using finite state machines, supporting multiple algorithms (pattern-based, WordPiece, SentencePiece Unigram LM, BPE) with prebuilt models for BERT, XLNET, GPT-2, and other NLP frameworks.

Yes, if you need fast tokenization for NLP inference. The package is stable, has no dependencies, and offers significant speed advantages over pure-Python alternatives. However, be aware it is dormant (last release September 2021) and may not receive updates for new Python versions or model formats. Suitable for production use where the existing models and algorithms meet your needs.

## Install

pip install blingfire
uv add blingfire
poetry add blingfire

## Installing blingfire

Before you install: Installation is straightforward with no runtime dependencies. The package is dormant (last release September 2021, no commits since December 2024), but the underlying C++ library is stable and widely used in production at Microsoft.

License in practice: Licensed under MIT (permissive), allowing free use, modification, and distribution with minimal restrictions.

Quickstart:

pip install blingfire

from blingfire import text_to_words, text_to_sentences

text = 'Hello world. This is a test.'
print(text_to_sentences(text))
print(text_to_words(text))

Verify before relying:

- Whether the package works with Python versions beyond 3.x (exact version support unspecified in metadata)
- Current maintenance status and whether dormancy affects compatibility with modern Python tooling

## Package facts

- License: not declared (permissive)
- Python support: unspecified
- Install friction: low
- Maintenance: dormant
- Downloads: 879.9K/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

fast tokenization nlp, bert wordpiece tokenizer, sentence segmentation, text tokenization library, finite state machine nlp, multilingual tokenization, sentencepiece alternative, tokenization, nlp, performance-critical

[View on SkillFed](https://skillfed.io/packages/blingfire) · [View on PyPI](https://pypi.org/project/blingfire/)