razdel

What it is and what it does

razdel is a rule-based tokenizer and sentence splitter designed specifically for Russian text. It provides two main functions: tokenize() breaks text into words and punctuation with character offsets, and sentenize() splits text into sentences. The package returns Substring objects that preserve the original text span, making it easy to map results back to the source.

The library is optimized against four Russian corpora (SynTagRus, OpenCorpora, GICRYA, RNC) consisting mainly of news and fiction. It trades off perfect accuracy for practical performance: the description acknowledges that tokenization has no single correct answer and documents its segmentation choices explicitly. Benchmarks show it achieves competitive error rates on these domains while maintaining reasonable speed.

Use it for:

• Preprocess Russian news articles or literary texts before feeding into downstream NLP models
• Extract individual words and punctuation from Russian text while preserving their exact positions for annotation or highlighting
• Split Russian documents into sentences for batch processing or analysis at the sentence level
• Build Russian text search or indexing pipelines that require accurate token boundaries
• Evaluate or compare tokenization quality on Russian corpora using the included benchmarking tools

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Splits Russian text into sentences and tokens using rule-based heuristics, returning character-offset substrings for precise text reconstruction.

Yes, if your text is primarily Russian news or fiction. The package is lightweight, actively maintained, permissively licensed, and offers competitive accuracy on its target domains. Install with caution if your text is from social media, legal documents, or scientific articles—the description explicitly warns performance may degrade outside news/fiction.

Install

razdel on PyPI

pip install razdel

uv add razdel

poetry add razdel

Installing razdel

Before you install

Low friction: pure Python wheel with no runtime dependencies. Actively maintained as of April 2026 with 286 repository stars.

License in practice

MIT license permits unrestricted use, modification, and distribution with minimal attribution requirements.

Quickstart

from razdel import tokenize, sentenize

tokens = list(tokenize('Кружка-термос на 0.5л'))
sents = list(sentenize('Так в чем же дело? Не радуют.'))

Verify before relying

• Whether performance on non-news/fiction domains (social media, legal, scientific) is documented or measurable
• Exact Python version support range beyond the '3.5+' claim in the description

Package facts

Evidence: razdel-0.5.0-py3-none-any.whl

Keywords: nlp, natural language processing, russian, token, sentence, tokenize

Tags

More Linguistic packages