{"categories":[{"label":"Linguistic","url":"https://skillfed.io/packages/category/text-processing-linguistic/2"}],"enrichment":{"capability":"Splits long text into semantically meaningful chunks sized for LLM context windows, respecting sentence and paragraph boundaries rather than breaking mid-thought.","skillfed_tags":["llm-tooling","document-processing","tokenization"],"use_cases":["Prepare long documents for retrieval-augmented generation (RAG) by splitting into token-bounded chunks that respect paragraph structure.","Split markdown documentation into semantic sections for indexing and search without breaking code blocks or inline formatting.","Chunk text for fine-tuning datasets where you need to respect sentence and paragraph boundaries to preserve training signal.","Prepare long articles or books for LLM summarization by splitting into context-window-sized pieces that maintain narrative coherence.","Build a document ingestion pipeline that respects both character/token limits and semantic structure for downstream NLP tasks."],"what_it_does":"semantic-text-splitter is a Python library that breaks long documents into smaller chunks optimized for LLM processing. Rather than splitting at fixed character boundaries, it respects semantic structure\u2014sentences, paragraphs, markdown blocks, and newline sequences\u2014to keep related content together. You can specify chunk size by character count, token range, or custom callback, and it supports multiple tokenizer backends (Hugging Face, Tiktoken) or plain character counting.\n\nThe library provides two main splitters: TextSplitter for plain text and MarkdownSplitter for markdown documents. It uses a hierarchical approach that tries to fill chunks to your target size while never breaking at lower semantic levels if a higher-level boundary is available. This is useful when preparing documents for RAG pipelines, prompt engineering, or any workflow where you need to feed text to models with fixed context limits while preserving meaning.","worth_installing":"Yes, if you need semantic-aware text chunking for LLM workflows. The library is actively maintained, has no known vulnerabilities, and offers a cleaner API than character-only splitting. The MIT license poses no restrictions. Medium install friction (Rust compilation) is a minor trade-off for the performance and semantic quality it provides. Install if you're building RAG systems, prompt pipelines, or document processing for language models."},"id":"semantic-text-splitter","links":{"html":"https://skillfed.io/packages/semantic-text-splitter","md":"https://skillfed.io/packages/semantic-text-splitter.md","pypi":"https://pypi.org/project/semantic-text-splitter/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-06-16","license_spdx":null,"license_treatment":"permissive","name":"semantic-text-splitter","python_support":"supports_current","summary":"Split text into semantic chunks, up to a desired chunk size. Supports calculating length by characters and tokens, and is callable from Rust and Python."},"popularity":{"monthly_downloads":307442,"position":7775,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.32.0"}