---
id: semantic-text-splitter
version: "0.32.0"
license: MIT
license_treatment: permissive
maintenance: active
---

# semantic-text-splitter — Split text into semantic chunks, up to a desired chunk size. Supports calculating length by characters and tokens, and is callable from Rust and Python.

License: permissive · Maintenance: active · Downloads: 307.4K/mo

## What it is and what it does

semantic-text-splitter is a Python library that breaks long documents into smaller chunks optimized for LLM processing. Rather than splitting at fixed character boundaries, it respects semantic structure—sentences, paragraphs, markdown blocks, and newline sequences—to keep related content together. You can specify chunk size by character count, token range, or custom callback, and it supports multiple tokenizer backends (Hugging Face, Tiktoken) or plain character counting.

The library provides two main splitters: TextSplitter for plain text and MarkdownSplitter for markdown documents. It uses a hierarchical approach that tries to fill chunks to your target size while never breaking at lower semantic levels if a higher-level boundary is available. This is useful when preparing documents for RAG pipelines, prompt engineering, or any workflow where you need to feed text to models with fixed context limits while preserving meaning.

Use it for:

- Prepare long documents for retrieval-augmented generation (RAG) by splitting into token-bounded chunks that respect paragraph structure.
- Split markdown documentation into semantic sections for indexing and search without breaking code blocks or inline formatting.
- Chunk text for fine-tuning datasets where you need to respect sentence and paragraph boundaries to preserve training signal.
- Prepare long articles or books for LLM summarization by splitting into context-window-sized pieces that maintain narrative coherence.
- Build a document ingestion pipeline that respects both character/token limits and semantic structure for downstream NLP tasks.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Splits long text into semantically meaningful chunks sized for LLM context windows, respecting sentence and paragraph boundaries rather than breaking mid-thought.

Yes, if you need semantic-aware text chunking for LLM workflows. The library is actively maintained, has no known vulnerabilities, and offers a cleaner API than character-only splitting. The MIT license poses no restrictions. Medium install friction (Rust compilation) is a minor trade-off for the performance and semantic quality it provides. Install if you're building RAG systems, prompt pipelines, or document processing for language models.

## Install

pip install semantic-text-splitter
uv add semantic-text-splitter
poetry add semantic-text-splitter

## Installing semantic-text-splitter

Before you install: Medium install friction due to compiled Rust bindings, but pre-built wheels cover common platforms (x86_64, ARM, Windows). Requires Python 3.10+. Active maintenance with recent releases.

License in practice: MIT license permits unrestricted use, modification, and distribution with only attribution required—no restrictions on commercial or proprietary use.

Quickstart:

from semantic_text_splitter import TextSplitter

splitter = TextSplitter(max_characters=1000)
chunks = splitter.chunks("your document text")

Requires Python 3.10 or later; compiled Rust wheels may not be available for all architectures.

Verify before relying:

- Performance characteristics (speed, memory overhead) compared to alternatives like LangChain's TextSplitter.
- Whether custom tokenizers beyond Hugging Face and Tiktoken are supported.
- Behavior when a single semantic unit exceeds the specified chunk size limit.

## Package facts

- License: MIT (permissive)
- Python support: supports_current
- Install friction: medium
- Maintenance: active
- Downloads: 307.4K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

text chunking for llm context, semantic text splitting, document chunking by tokens, split text into chunks, nlp text segmentation, markdown document splitting, tokenizer-aware text splitting, llm-tooling, document-processing, tokenization

[View on SkillFed](https://skillfed.io/packages/semantic-text-splitter) · [View on PyPI](https://pypi.org/project/semantic-text-splitter/)