{"categories":[{"label":"Libraries","url":"https://skillfed.io/packages/category/software-development-libraries/4"},{"label":"Text Processing","url":"https://skillfed.io/packages/category/text-processing/2"},{"label":"Information Analysis","url":"https://skillfed.io/packages/category/scientific-engineering-information-analysis"},{"label":"Linguistic","url":"https://skillfed.io/packages/category/text-processing-linguistic"}],"enrichment":{"capability":"Splits Indo-European text into sentences and words using rule-based segmentation and tokenization, with command-line tools for batch processing.","skillfed_tags":["sentence-segmentation","tokenization","text-preprocessing"],"use_cases":["Preprocessing text corpora for NLP pipelines that require sentence and word-level boundaries before downstream processing","Batch processing plain-text documents via command-line to normalize and split text for indexing or analysis","Extracting tokens from multilingual documents where Indo-European language support is sufficient and rule-based segmentation is preferred","Splitting English text with contractions and possessives using built-in pattern matching and splitting functions","Normalizing line breaks and sentence boundaries in documents with irregular formatting before further processing"],"what_it_does":"segtok provides two core modules for breaking down text: segtok.segmenter splits text into sentences, and segtok.tokenizer breaks sentences into words and symbols. It is designed for Indo-European languages (English, Spanish, German) and includes command-line tools for processing plain-text files. The package depends only on regex and installs with low friction.\n\nThe segmenter handles sentence boundaries including abbreviations, numbers, and edge cases like terminals followed by invalid characters. The tokenizer offers multiple strategies, with web_tokenizer providing semantic splitting while preserving URLs and email addresses. It also includes utilities for handling English contractions and possessive markers. However, the package has been abandoned since 2021-12-15, and the description explicitly recommends syntok as a successor that fixes tricky splitting issues.","worth_installing":"No\u2014the description explicitly recommends syntok as a successor that fixes known issues with sentence splitting. segtok itself is abandoned (last commit 2021-12-15), and while it has low install friction and no known vulnerabilities, choosing an unmaintained package when a maintained successor exists is not justified."},"id":"segtok","links":{"html":"https://skillfed.io/packages/segtok","md":"https://skillfed.io/packages/segtok.md","pypi":"https://pypi.org/project/segtok/"},"maintenance":{"status":"abandoned"},"meta":{"latest_release":"2021-12-15","license_spdx":null,"license_treatment":"permissive","name":"segtok","python_support":"unspecified","summary":"sentence segmentation and word tokenization tools"},"popularity":{"monthly_downloads":767397,"position":5116,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"1.5.11"}