segments
Segmentation with orthography profiles
Decision gist · record as of 2026-08-14
Yes. Segments is actively maintained, has low install friction, carries no known vulnerabilities, and is licensed permissively. Install it if you need Unicode-aware tokenization with orthography profile support for linguistic or text-processing work; skip it if you only need basic whitespace or regex tokenization.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.9 or later.
- Low install friction with a pure-Python wheel distribution.
- Actively maintained with recent commits and stable production status across Python 3.9–3.14.
License · maintenance · safety
Apache 2.0 (permissive) — Apache 2.0 is permissive; you can use, modify, and distribute this package freely in commercial and private projects with minimal restrictions.
last release 2026-03-07 (160 days) · last repo commit 2026-03-07 · 41 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 1,377,896 downloads/mo, #3,984 on PyPI
Alternatives
Verify before relying
pip install segments
from segments import Tokenizer
t = Tokenizer()
print(t('abcd')) # 'a b c d'- Whether the regex and csvw dependencies introduce any notable security or maintenance concerns beyond what the fact sheet shows.
- Real-world performance characteristics when tokenizing large texts or with complex orthography profiles.
What it is and what it does
Segments is a Unicode tokenization library that splits text into meaningful units—graphemes, words, or custom segments—based on orthography profiles. It implements the linear algorithm from the Unicode Cookbook's orthography profile specification, allowing you to define how text should be segmented through a profile file that maps grapheme sequences to canonical forms or other output columns.
The package works both as a command-line tool and a Python API. You can build a profile from sample text, edit it to define custom segmentation rules, and then apply it to tokenize new text. It depends on regex for pattern matching and csvw for profile file handling, making it suitable for linguistic analysis, text preprocessing, and language-specific tokenization tasks where Unicode grapheme clusters or custom orthographic rules matter.
Use it for
- Tokenize text in languages with complex orthographies or diacritics where standard whitespace splitting is insufficient.
- Build and apply custom orthography profiles to normalize or map grapheme sequences in linguistic research or NLP pipelines.
- Preprocess text for phonetic or morphological analysis by defining segment boundaries according to linguistic conventions.
- Extract and analyze grapheme frequency and distribution from text samples via the command-line profile tool.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
Segments is actively maintained, has low install friction, carries no known vulnerabilities, and is licensed permissively. Install it if you need Unicode-aware tokenization with orthography profile support for linguistic or text-processing work; skip it if you only need basic whitespace or regex tokenization.
Install
segments on PyPI
Before you install
Low install friction with a pure-Python wheel distribution. Actively maintained with recent commits and stable production status across Python 3.9–3.14.
Requires Python 3.9 or later.
License in practice
Apache 2.0 is permissive; you can use, modify, and distribute this package freely in commercial and private projects with minimal restrictions.
Quickstart
pip install segments
from segments import Tokenizer
t = Tokenizer()
print(t('abcd')) # 'a b c d'Verify before relying
- Whether the regex and csvw dependencies introduce any notable security or maintenance concerns beyond what the fact sheet shows.
- Real-world performance characteristics when tokenizing large texts or with complex orthography profiles.
Package facts
| License | Apache 2.0 permissive |
| Python support | Supports the current Python release >=3.9 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 2 packagesregexcsvw |
| Maintenance | Actively maintained 160 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 1,377,896 / month, #3,984 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 5 - Production/StableIntended Audience :: DevelopersIntended Audience :: Science/ResearchLicense :: OSI Approved :: Apache Software LicenseNatural Language :: EnglishOperating System :: OS IndependentProgramming Language :: Python :: 3Programming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Programming Language :: Python :: 3.9Programming Language :: Python :: Implementation :: CPythonProgramming Language :: Python :: Implementation :: PyPy |
Evidence: segments-2.4.0-py2.py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “unicode tokenization”
- segmentsSegments provides Unicode-aware tokenization and orthography-based…
- tensorflow-textTensorFlow Text provides text preprocessing operations and tokenizers…
- sentencepieceSentencePiece is an unsupervised text tokenizer and detokenizer that…
Give your agent the search over MCP, or paste the wish link into any chat.
More Linguistic packages
Detects and normalizes text encoding from unknown or ambiguous sources, supporting all IANA character sets that Python's core library provides codecs for, with the ability to register custom codecs.
tiktoken is a fast BPE tokenizer that converts text into token sequences compatible with OpenAI models, supporting multiple encoding schemes including o200k_base and model-specific encodings.
Install it if you work with OpenAI APIs or need to understand token boundaries in GPT-family models.
Detects character encoding and language in byte sequences with high accuracy, supporting 99 encodings and returning confidence scores, language tags, and MIME types.
Install it if you need to detect character encoding or language in byte data; the rewrite makes it substantially faster and more accurate than its predecessors.
Converts Unicode text to ASCII by transliterating non-ASCII characters into their closest ASCII equivalents, with no runtime dependencies.
However, if transliteration quality or ongoing maintenance matters, consider unidecode instead despite its GPL-only license.
Lark is a parsing library that builds abstract syntax trees from context-free grammars, supporting multiple parsing algorithms (Earley, LALR(1), CYK) with automatic line and column tracking.
Python bindings to the tree-sitter parsing library, enabling incremental parsing and syntax tree analysis for source code.
See also uniseg · tokenizer · unicode-segmentation-rs · sentencepiece · segtok · jieba3k · razdel · sacremoses · wordsegment · jieba