semchunk

What it is and what it does

semchunk is a Python library that breaks text into smaller, semantically coherent chunks—a critical preprocessing step for retrieval-augmented generation (RAG) and other NLP workflows. It uses a hierarchical chunking algorithm to preserve local semantic context better than simpler splitting strategies. The library is agnostic about tokenization: you can supply any tokenizer (Tiktoken, Hugging Face Transformers, or a custom function) or a simple token counter, and semchunk will respect your token budget while keeping related text together.

The library supports chunk overlapping (by ratio or absolute token count), offset tracking (to map chunks back to source positions), and multiprocessing for batch operations. Optionally, you can enable AI-powered chunking by providing an Isaacus enrichment model name and API key, which uses semantic understanding to make smarter split decisions. It requires only dill and tqdm as dependencies, installs as a pure Python wheel, and supports Python 3.10 through 3.14.

Use it for:

• Prepare documents for RAG pipelines by splitting them into token-bounded chunks that preserve semantic coherence for embedding and retrieval.
• Batch-process large text corpora with multiprocessing, tracking chunk offsets to reconstruct source positions after retrieval.
• Integrate custom tokenizers (e.g., domain-specific or model-specific) into a chunking workflow without rewriting splitting logic.
• Overlap chunks for sliding-window context in language model fine-tuning or evaluation tasks.
• Use AI-powered chunking (via Isaacus) to make semantic split decisions for complex documents like legal or scientific texts.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

semchunk splits text into semantically meaningful chunks while preserving local context, supporting custom tokenizers, chunk overlapping, offsets, and optional AI-powered chunking via the Isaacus API.

Yes. semchunk is production-ready, actively maintained, has no known vulnerabilities, and solves a real problem (semantic text chunking) with low install friction and a permissive license. It is widely used (top 5000 PyPI packages) and offers flexibility via custom tokenizers and optional AI enhancement. Install it if you need to chunk text for RAG, embeddings, or language model workflows.

Install

semchunk on PyPI

pip install semchunk

uv add semchunk

poetry add semchunk

Installing semchunk

Before you install

Low friction: pure Python wheel with only dill and tqdm as runtime dependencies. Actively maintained with recent releases; last commit 2026-06-13. Marked production-ready and used in Docling and the Microsoft Intelligence Toolkit.

License in practice

MIT license (permissive): you can use, modify, and distribute semchunk freely in commercial and private projects without restriction, provided you include the license notice.

Quickstart

pip install semchunk

import semchunk

chunker = semchunk.chunkerify(lambda text: len(text.split()), chunk_size=4)
chunks = chunker('The quick brown fox jumps over the lazy dog.')
print(chunks)  # ['The quick brown fox', 'jumps over the', 'lazy dog.']

Requires Python 3.10 or later. For AI-powered chunking, the Isaacus SDK and a valid ISAACUS_API_KEY environment variable are required.

Verify before relying

• Whether the claimed 15% RAG performance improvement over competitors is independently verified or from internal benchmarks.
• Current scale of 'millions of times per month' downloads and whether this refers to semchunk specifically or includes transitive installs.
• Specific performance characteristics (speed, memory usage) for large documents or high-concurrency scenarios.

Package facts

Evidence: semchunk-4.1.1-py3-none-any.whl

Keywords: ai, chunk, chunker, chunking, chunks, nlp, split, splits, splitter, splitting, text

Tags

More Python Modules packages