{"categories":[{"label":"Linguistic","url":"https://skillfed.io/packages/category/text-processing-linguistic/2"}],"enrichment":{"capability":"Splits unsegmented English text into individual words using a corpus-based statistical approach, handling cases where spaces have been removed or are ambiguous.","skillfed_tags":["nlp","text-processing","unmaintained"],"use_cases":["Recover readable words from concatenated text like 'thequickbrownfox' in OCR or data-cleaning pipelines.","Process hashtags or domain names where word boundaries are unclear, splitting '#pythonrocks' into candidate words.","Batch-segment large files of unsegmented English text via the command-line interface.","Analyze corpus frequency data directly (UNIGRAMS and BIGRAMS dictionaries) for linguistic research or spelling preference studies."],"what_it_does":"WordSegment is a pure-Python library that reconstructs word boundaries in English text where spaces have been removed or are ambiguous. It uses unigram and bigram frequency data derived from the Google Web Trillion Word Corpus to score candidate segmentations and select the most likely split. The package includes 333,000 unigrams and 250,000 bigrams, all lowercased with punctuation removed, and provides both a programmatic API and a command-line interface for batch processing.\n\nThe library works by loading corpus frequency tables into memory and then using dynamic programming to find the segmentation that maximizes the product of word frequencies. It handles the canonical form of input (lowercasing, punctuation removal) automatically via a clean() function. Maximum word length is 24 characters, matching the corpus data. The package has no external dependencies and runs on both Python 2 and 3, though it has not been updated since 2018.","worth_installing":"Yes, if you need English word segmentation and can tolerate an unmaintained package. The library is stable, has no dependencies, and works reliably for its narrow use case. However, the 2960-day gap since the last release means no updates for modern Python versions, security patches, or corpus improvements. Install only if the 2018-era corpus data and Python 3.6 compatibility are sufficient for your task."},"id":"wordsegment","links":{"html":"https://skillfed.io/packages/wordsegment","md":"https://skillfed.io/packages/wordsegment.md","pypi":"https://pypi.org/project/wordsegment/"},"maintenance":{"status":"abandoned"},"meta":{"latest_release":"2018-07-07","license_spdx":null,"license_treatment":"permissive","name":"wordsegment","python_support":"unspecified","summary":"English word segmentation."},"popularity":{"monthly_downloads":123744,"position":11901,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"1.3.1"}