{"categories":[{"label":"Linguistic","url":"https://skillfed.io/packages/category/text-processing-linguistic/2"}],"enrichment":{"capability":"Splits Russian text into sentences and tokens using rule-based heuristics, returning character-offset substrings for precise text reconstruction.","skillfed_tags":["russian-nlp","rule-based","tokenization"],"use_cases":["Preprocess Russian news articles or literary texts before feeding into downstream NLP models","Extract individual words and punctuation from Russian text while preserving their exact positions for annotation or highlighting","Split Russian documents into sentences for batch processing or analysis at the sentence level","Build Russian text search or indexing pipelines that require accurate token boundaries","Evaluate or compare tokenization quality on Russian corpora using the included benchmarking tools"],"what_it_does":"razdel is a rule-based tokenizer and sentence splitter designed specifically for Russian text. It provides two main functions: tokenize() breaks text into words and punctuation with character offsets, and sentenize() splits text into sentences. The package returns Substring objects that preserve the original text span, making it easy to map results back to the source.\n\nThe library is optimized against four Russian corpora (SynTagRus, OpenCorpora, GICRYA, RNC) consisting mainly of news and fiction. It trades off perfect accuracy for practical performance: the description acknowledges that tokenization has no single correct answer and documents its segmentation choices explicitly. Benchmarks show it achieves competitive error rates on these domains while maintaining reasonable speed.","worth_installing":"Yes, if your text is primarily Russian news or fiction. The package is lightweight, actively maintained, permissively licensed, and offers competitive accuracy on its target domains. Install with caution if your text is from social media, legal documents, or scientific articles\u2014the description explicitly warns performance may degrade outside news/fiction."},"id":"razdel","links":{"html":"https://skillfed.io/packages/razdel","md":"https://skillfed.io/packages/razdel.md","pypi":"https://pypi.org/project/razdel/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2020-03-26","license_spdx":null,"license_treatment":"permissive","name":"razdel","python_support":"unspecified","summary":"Splits russian text into tokens, sentences, section. Rule-based"},"popularity":{"monthly_downloads":130482,"position":11639,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.5.0"}