segtok

What it is and what it does

segtok provides two core modules for breaking down text: segtok.segmenter splits text into sentences, and segtok.tokenizer breaks sentences into words and symbols. It is designed for Indo-European languages (English, Spanish, German) and includes command-line tools for processing plain-text files. The package depends only on regex and installs with low friction.

The segmenter handles sentence boundaries including abbreviations, numbers, and edge cases like terminals followed by invalid characters. The tokenizer offers multiple strategies, with web_tokenizer providing semantic splitting while preserving URLs and email addresses. It also includes utilities for handling English contractions and possessive markers. However, the package has been abandoned since 2021-12-15, and the description explicitly recommends syntok as a successor that fixes tricky splitting issues.

Use it for:

• Preprocessing text corpora for NLP pipelines that require sentence and word-level boundaries before downstream processing
• Batch processing plain-text documents via command-line to normalize and split text for indexing or analysis
• Extracting tokens from multilingual documents where Indo-European language support is sufficient and rule-based segmentation is preferred
• Splitting English text with contractions and possessives using built-in pattern matching and splitting functions
• Normalizing line breaks and sentence boundaries in documents with irregular formatting before further processing

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Splits Indo-European text into sentences and words using rule-based segmentation and tokenization, with command-line tools for batch processing.

No—the description explicitly recommends syntok as a successor that fixes known issues with sentence splitting. segtok itself is abandoned (last commit 2021-12-15), and while it has low install friction and no known vulnerabilities, choosing an unmaintained package when a maintained successor exists is not justified.

Install

segtok on PyPI

pip install segtok

uv add segtok

poetry add segtok

Installing segtok

Before you install

Low friction install with a single regex dependency. However, the package is abandoned—last release was 2021-12-15 and no commits since.

License in practice

MIT license is permissive and imposes no restrictions on use, modification, or distribution in proprietary or open-source projects.

Quickstart

pip install segtok

from segtok.segmenter import split_sentences
from segtok.tokenizer import web_tokenizer

text = "Hello world. This is a test."
for sentence in split_sentences(text):
    tokens = list(web_tokenizer(sentence))
    print(tokens)

The regex dependency requires python-dev or python3-dev headers on Linux systems to compile.

Verify before relying

• Whether the known issues mentioned in the description (sentence splitting with terminals not followed by spaces) affect your use case
• Current compatibility with Python versions beyond 3.8, given the package is abandoned and classifiers list only up to 3.8

Package facts

Evidence: segtok-1.5.11-py3-none-any.whl

Keywords: sentence, segmenter, splitter, split, word, tokenizer, token

Tags

More Libraries packages