segments

What it is and what it does

Segments is a Unicode tokenization library that splits text into meaningful units—graphemes, words, or custom segments—based on orthography profiles. It implements the linear algorithm from the Unicode Cookbook's orthography profile specification, allowing you to define how text should be segmented through a profile file that maps grapheme sequences to canonical forms or other output columns.

The package works both as a command-line tool and a Python API. You can build a profile from sample text, edit it to define custom segmentation rules, and then apply it to tokenize new text. It depends on regex for pattern matching and csvw for profile file handling, making it suitable for linguistic analysis, text preprocessing, and language-specific tokenization tasks where Unicode grapheme clusters or custom orthographic rules matter.

Use it for:

• Tokenize text in languages with complex orthographies or diacritics where standard whitespace splitting is insufficient.
• Build and apply custom orthography profiles to normalize or map grapheme sequences in linguistic research or NLP pipelines.
• Preprocess text for phonetic or morphological analysis by defining segment boundaries according to linguistic conventions.
• Extract and analyze grapheme frequency and distribution from text samples via the command-line profile tool.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Segments provides Unicode-aware tokenization and orthography-based text segmentation, using orthography profiles to define how text should be split into meaningful units.

Yes. Segments is actively maintained, has low install friction, carries no known vulnerabilities, and is licensed permissively. Install it if you need Unicode-aware tokenization with orthography profile support for linguistic or text-processing work; skip it if you only need basic whitespace or regex tokenization.

Install

segments on PyPI

pip install segments

uv add segments

poetry add segments

Installing segments

Before you install

Low install friction with a pure-Python wheel distribution. Actively maintained with recent commits and stable production status across Python 3.9–3.14.

License in practice

Apache 2.0 is permissive; you can use, modify, and distribute this package freely in commercial and private projects with minimal restrictions.

Quickstart

pip install segments

from segments import Tokenizer
t = Tokenizer()
print(t('abcd'))  # 'a b c d'

Requires Python 3.9 or later.

Verify before relying

• Whether the regex and csvw dependencies introduce any notable security or maintenance concerns beyond what the fact sheet shows.
• Real-world performance characteristics when tokenizing large texts or with complex orthography profiles.

Package facts

Evidence: segments-2.4.0-py2.py3-none-any.whl

Keywords: linguistics, tokenizer

Tags

More Linguistic packages