---
id: semchunk
version: "4.1.1"
license: MIT
license_treatment: permissive
maintenance: active
---

# semchunk — A Python library for splitting text into smaller chunks while preserving as much local semantic context as possible.

License: permissive · Maintenance: active · Downloads: 3.9M/mo

## What it is and what it does

semchunk is a Python library that breaks text into smaller, semantically coherent chunks—a critical preprocessing step for retrieval-augmented generation (RAG) and other NLP workflows. It uses a hierarchical chunking algorithm to preserve local semantic context better than simpler splitting strategies. The library is agnostic about tokenization: you can supply any tokenizer (Tiktoken, Hugging Face Transformers, or a custom function) or a simple token counter, and semchunk will respect your token budget while keeping related text together.

The library supports chunk overlapping (by ratio or absolute token count), offset tracking (to map chunks back to source positions), and multiprocessing for batch operations. Optionally, you can enable AI-powered chunking by providing an Isaacus enrichment model name and API key, which uses semantic understanding to make smarter split decisions. It requires only dill and tqdm as dependencies, installs as a pure Python wheel, and supports Python 3.10 through 3.14.

Use it for:

- Prepare documents for RAG pipelines by splitting them into token-bounded chunks that preserve semantic coherence for embedding and retrieval.
- Batch-process large text corpora with multiprocessing, tracking chunk offsets to reconstruct source positions after retrieval.
- Integrate custom tokenizers (e.g., domain-specific or model-specific) into a chunking workflow without rewriting splitting logic.
- Overlap chunks for sliding-window context in language model fine-tuning or evaluation tasks.
- Use AI-powered chunking (via Isaacus) to make semantic split decisions for complex documents like legal or scientific texts.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

semchunk splits text into semantically meaningful chunks while preserving local context, supporting custom tokenizers, chunk overlapping, offsets, and optional AI-powered chunking via the Isaacus API.

Yes. semchunk is production-ready, actively maintained, has no known vulnerabilities, and solves a real problem (semantic text chunking) with low install friction and a permissive license. It is widely used (top 5000 PyPI packages) and offers flexibility via custom tokenizers and optional AI enhancement. Install it if you need to chunk text for RAG, embeddings, or language model workflows.

## Install

pip install semchunk
uv add semchunk
poetry add semchunk

## Installing semchunk

Before you install: Low friction: pure Python wheel with only dill and tqdm as runtime dependencies. Actively maintained with recent releases; last commit 2026-06-13. Marked production-ready and used in Docling and the Microsoft Intelligence Toolkit.

License in practice: MIT license (permissive): you can use, modify, and distribute semchunk freely in commercial and private projects without restriction, provided you include the license notice.

Quickstart:

pip install semchunk

import semchunk

chunker = semchunk.chunkerify(lambda text: len(text.split()), chunk_size=4)
chunks = chunker('The quick brown fox jumps over the lazy dog.')
print(chunks)  # ['The quick brown fox', 'jumps over the', 'lazy dog.']

Requires Python 3.10 or later. For AI-powered chunking, the Isaacus SDK and a valid ISAACUS_API_KEY environment variable are required.

Verify before relying:

- Whether the claimed 15% RAG performance improvement over competitors is independently verified or from internal benchmarks.
- Current scale of 'millions of times per month' downloads and whether this refers to semchunk specifically or includes transitive installs.
- Specific performance characteristics (speed, memory usage) for large documents or high-concurrency scenarios.

## Package facts

- License: MIT (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 3.9M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

text chunking semantic context, split text for RAG embeddings, token-aware text splitting, hierarchical document chunking, chunk overlap and offsets, custom tokenizer support, AI-powered text segmentation, rag-preprocessing, text-splitting, semantic-chunking

[View on SkillFed](https://skillfed.io/packages/semchunk) · [View on PyPI](https://pypi.org/project/semchunk/)