wordsegment

What it is and what it does

WordSegment is a pure-Python library that reconstructs word boundaries in English text where spaces have been removed or are ambiguous. It uses unigram and bigram frequency data derived from the Google Web Trillion Word Corpus to score candidate segmentations and select the most likely split. The package includes 333,000 unigrams and 250,000 bigrams, all lowercased with punctuation removed, and provides both a programmatic API and a command-line interface for batch processing.

The library works by loading corpus frequency tables into memory and then using dynamic programming to find the segmentation that maximizes the product of word frequencies. It handles the canonical form of input (lowercasing, punctuation removal) automatically via a clean() function. Maximum word length is 24 characters, matching the corpus data. The package has no external dependencies and runs on both Python 2 and 3, though it has not been updated since 2018.

Use it for:

• Recover readable words from concatenated text like 'thequickbrownfox' in OCR or data-cleaning pipelines.
• Process hashtags or domain names where word boundaries are unclear, splitting '#pythonrocks' into candidate words.
• Batch-segment large files of unsegmented English text via the command-line interface.
• Analyze corpus frequency data directly (UNIGRAMS and BIGRAMS dictionaries) for linguistic research or spelling preference studies.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Splits unsegmented English text into individual words using a corpus-based statistical approach, handling cases where spaces have been removed or are ambiguous.

Yes, if you need English word segmentation and can tolerate an unmaintained package. The library is stable, has no dependencies, and works reliably for its narrow use case. However, the 2960-day gap since the last release means no updates for modern Python versions, security patches, or corpus improvements. Install only if the 2018-era corpus data and Python 3.6 compatibility are sufficient for your task.

Install

wordsegment on PyPI

pip install wordsegment

uv add wordsegment

poetry add wordsegment

Installing wordsegment

Before you install

Installation is straightforward with no runtime dependencies. However, the package has been abandoned since 2018 (2960 days without release) and is no longer maintained, so expect no bug fixes or updates.

License in practice

Licensed under Apache 2.0 (permissive), allowing commercial and private use with minimal restrictions—you may use, modify, and distribute freely provided you include the license notice.

Quickstart

pip install wordsegment

from wordsegment import load, segment
load()
result = segment('thisisatest')
print(result)  # ['this', 'is', 'a', 'test']

The load() function must be called once to read corpus data from disk before segmenting; this is a one-time setup cost per process.

Verify before relying

• Whether the corpus data (333,000 unigrams, 250,000 bigrams) remains adequate for modern English or specialized domains.
• Performance characteristics on very long texts or high-volume batch processing.
• Compatibility with Python versions beyond 3.6, given the package's age and lack of maintenance.

Package facts

Evidence: wordsegment-1.3.1-py2.py3-none-any.whl

Tags

More Linguistic packages