semantic-text-splitter

What it is and what it does

semantic-text-splitter is a Python library that breaks long documents into smaller chunks optimized for LLM processing. Rather than splitting at fixed character boundaries, it respects semantic structure—sentences, paragraphs, markdown blocks, and newline sequences—to keep related content together. You can specify chunk size by character count, token range, or custom callback, and it supports multiple tokenizer backends (Hugging Face, Tiktoken) or plain character counting.

The library provides two main splitters: TextSplitter for plain text and MarkdownSplitter for markdown documents. It uses a hierarchical approach that tries to fill chunks to your target size while never breaking at lower semantic levels if a higher-level boundary is available. This is useful when preparing documents for RAG pipelines, prompt engineering, or any workflow where you need to feed text to models with fixed context limits while preserving meaning.

Use it for:

• Prepare long documents for retrieval-augmented generation (RAG) by splitting into token-bounded chunks that respect paragraph structure.
• Split markdown documentation into semantic sections for indexing and search without breaking code blocks or inline formatting.
• Chunk text for fine-tuning datasets where you need to respect sentence and paragraph boundaries to preserve training signal.
• Prepare long articles or books for LLM summarization by splitting into context-window-sized pieces that maintain narrative coherence.
• Build a document ingestion pipeline that respects both character/token limits and semantic structure for downstream NLP tasks.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Splits long text into semantically meaningful chunks sized for LLM context windows, respecting sentence and paragraph boundaries rather than breaking mid-thought.

Yes, if you need semantic-aware text chunking for LLM workflows. The library is actively maintained, has no known vulnerabilities, and offers a cleaner API than character-only splitting. The MIT license poses no restrictions. Medium install friction (Rust compilation) is a minor trade-off for the performance and semantic quality it provides. Install if you're building RAG systems, prompt pipelines, or document processing for language models.

Install

semantic-text-splitter on PyPI

pip install semantic-text-splitter

uv add semantic-text-splitter

poetry add semantic-text-splitter

Installing semantic-text-splitter

Before you install

Medium install friction due to compiled Rust bindings, but pre-built wheels cover common platforms (x86_64, ARM, Windows). Requires Python 3.10+. Active maintenance with recent releases.

License in practice

MIT license permits unrestricted use, modification, and distribution with only attribution required—no restrictions on commercial or proprietary use.

Quickstart

from semantic_text_splitter import TextSplitter

splitter = TextSplitter(max_characters=1000)
chunks = splitter.chunks("your document text")

Requires Python 3.10 or later; compiled Rust wheels may not be available for all architectures.

Verify before relying

• Performance characteristics (speed, memory overhead) compared to alternatives like LangChain's TextSplitter.
• Whether custom tokenizers beyond Hugging Face and Tiktoken are supported.
• Behavior when a single semantic unit exceeds the specified chunk size limit.

Package facts

Evidence: semantic_text_splitter-0.32.0-cp310-abi3-macosx_10_12_x86_64.whl; semantic_text_splitter-0.32.0-cp310-abi3-macosx_11_0_arm64.whl; semantic_text_splitter-0.32.0-cp310-abi3-manylinux_2_28_aarch64.whl; semantic_text_splitter-0.32.0-cp310-abi3-manylinux_2_28_armv7l.whl; semantic_text_splitter-0.32.0-cp310-abi3-manylinux_2_28_ppc64le.whl; semantic_text_splitter-0.32.0-cp310-abi3-manylinux_2_28_s390x.whl; semantic_text_splitter-0.32.0-cp310-abi3-manylinux_2_28_x86_64.whl; semantic_text_splitter-0.32.0-cp310-abi3-win32.whl; semantic_text_splitter-0.32.0-cp310-abi3-win_amd64.whl; semantic_text_splitter-0.32.0-cp314-cp314t-macosx_10_12_x86_64.whl; semantic_text_splitter-0.32.0-cp314-cp314t-macosx_11_0_arm64.whl; semantic_text_splitter-0.32.0-cp314-cp314t-manylinux_2_28_aarch64.whl; semantic_text_splitter-0.32.0-cp314-cp314t-manylinux_2_28_armv7l.whl; semantic_text_splitter-0.32.0-cp314-cp314t-manylinux_2_28_ppc64le.whl; semantic_text_splitter-0.32.0-cp314-cp314t-manylinux_2_28_s390x.whl; semantic_text_splitter-0.32.0-cp314-cp314t-manylinux_2_28_x86_64.whl; semantic_text_splitter-0.32.0-cp314-cp314t-win32.whl; semantic_text_splitter-0.32.0-cp314-cp314t-win_amd64.whl; semantic_text_splitter-0.32.0-cp314-cp314-win32.whl; semantic_text_splitter-0.32.0-cp314-cp314-win_amd64.whl

Keywords: text, split, tokenizer, nlp, ai

Tags

More Linguistic packages

Further reading