{"categories":[{"label":"Linguistic","url":"https://skillfed.io/packages/category/text-processing-linguistic"}],"enrichment":{"capability":"Segments provides Unicode-aware tokenization and orthography-based text segmentation, using orthography profiles to define how text should be split into meaningful units.","skillfed_tags":["unicode","nlp","linguistics"],"use_cases":["Tokenize text in languages with complex orthographies or diacritics where standard whitespace splitting is insufficient.","Build and apply custom orthography profiles to normalize or map grapheme sequences in linguistic research or NLP pipelines.","Preprocess text for phonetic or morphological analysis by defining segment boundaries according to linguistic conventions.","Extract and analyze grapheme frequency and distribution from text samples via the command-line profile tool."],"what_it_does":"Segments is a Unicode tokenization library that splits text into meaningful units\u2014graphemes, words, or custom segments\u2014based on orthography profiles. It implements the linear algorithm from the Unicode Cookbook's orthography profile specification, allowing you to define how text should be segmented through a profile file that maps grapheme sequences to canonical forms or other output columns.\n\nThe package works both as a command-line tool and a Python API. You can build a profile from sample text, edit it to define custom segmentation rules, and then apply it to tokenize new text. It depends on regex for pattern matching and csvw for profile file handling, making it suitable for linguistic analysis, text preprocessing, and language-specific tokenization tasks where Unicode grapheme clusters or custom orthographic rules matter.","worth_installing":"Yes. Segments is actively maintained, has low install friction, carries no known vulnerabilities, and is licensed permissively. Install it if you need Unicode-aware tokenization with orthography profile support for linguistic or text-processing work; skip it if you only need basic whitespace or regex tokenization."},"id":"segments","links":{"html":"https://skillfed.io/packages/segments","md":"https://skillfed.io/packages/segments.md","pypi":"https://pypi.org/project/segments/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-03-07","license_spdx":null,"license_treatment":"permissive","name":"segments","python_support":"supports_current","summary":"Segmentation with orthography profiles"},"popularity":{"monthly_downloads":1377896,"position":3984,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"2.4.0"}