semchunk
A Python library for splitting text into smaller chunks while preserving as much local semantic context as possible.
Decision gist · record as of 2026-08-14
Yes. semchunk is production-ready, actively maintained, has no known vulnerabilities, and solves a real problem (semantic text chunking) with low install friction and a permissive license. It is widely used (top 5000 PyPI packages) and offers flexibility via custom tokenizers and optional AI enhancement. Install it if you need to chunk text for RAG, embeddings, or language model workflows.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.10 or later.
- For AI-powered chunking, the Isaacus SDK and a valid ISAACUS_API_KEY environment variable are required.
- Low friction: pure Python wheel with only dill and tqdm as runtime dependencies.
License · maintenance · safety
MIT (permissive) — MIT license (permissive): you can use, modify, and distribute semchunk freely in commercial and private projects without restriction, provided you include the license notice.
last release 2026-06-13 (62 days) · last repo commit 2026-06-13 · 661 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 3,936,290 downloads/mo, #2,444 on PyPI
Alternatives
Verify before relying
pip install semchunk
import semchunk
chunker = semchunk.chunkerify(lambda text: len(text.split()), chunk_size=4)
chunks = chunker('The quick brown fox jumps over the lazy dog.')
print(chunks) # ['The quick brown fox', 'jumps over the', 'lazy dog.']- Whether the claimed 15% RAG performance improvement over competitors is independently verified or from internal benchmarks.
- Current scale of 'millions of times per month' downloads and whether this refers to semchunk specifically or includes transitive installs.
- Specific performance characteristics (speed, memory usage) for large documents or high-concurrency scenarios.
What it is and what it does
semchunk is a Python library that breaks text into smaller, semantically coherent chunks—a critical preprocessing step for retrieval-augmented generation (RAG) and other NLP workflows. It uses a hierarchical chunking algorithm to preserve local semantic context better than simpler splitting strategies. The library is agnostic about tokenization: you can supply any tokenizer (Tiktoken, Hugging Face Transformers, or a custom function) or a simple token counter, and semchunk will respect your token budget while keeping related text together.
The library supports chunk overlapping (by ratio or absolute token count), offset tracking (to map chunks back to source positions), and multiprocessing for batch operations. Optionally, you can enable AI-powered chunking by providing an Isaacus enrichment model name and API key, which uses semantic understanding to make smarter split decisions. It requires only dill and tqdm as dependencies, installs as a pure Python wheel, and supports Python 3.10 through 3.14.
Use it for
- Prepare documents for RAG pipelines by splitting them into token-bounded chunks that preserve semantic coherence for embedding and retrieval.
- Batch-process large text corpora with multiprocessing, tracking chunk offsets to reconstruct source positions after retrieval.
- Integrate custom tokenizers (e.g., domain-specific or model-specific) into a chunking workflow without rewriting splitting logic.
- Overlap chunks for sliding-window context in language model fine-tuning or evaluation tasks.
- Use AI-powered chunking (via Isaacus) to make semantic split decisions for complex documents like legal or scientific texts.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
semchunk is production-ready, actively maintained, has no known vulnerabilities, and solves a real problem (semantic text chunking) with low install friction and a permissive license. It is widely used (top 5000 PyPI packages) and offers flexibility via custom tokenizers and optional AI enhancement. Install it if you need to chunk text for RAG, embeddings, or language model workflows.
Install
semchunk on PyPI
Before you install
Low friction: pure Python wheel with only dill and tqdm as runtime dependencies. Actively maintained with recent releases; last commit 2026-06-13. Marked production-ready and used in Docling and the Microsoft Intelligence Toolkit.
Requires Python 3.10 or later. For AI-powered chunking, the Isaacus SDK and a valid ISAACUS_API_KEY environment variable are required.
License in practice
MIT license (permissive): you can use, modify, and distribute semchunk freely in commercial and private projects without restriction, provided you include the license notice.
Quickstart
pip install semchunk
import semchunk
chunker = semchunk.chunkerify(lambda text: len(text.split()), chunk_size=4)
chunks = chunker('The quick brown fox jumps over the lazy dog.')
print(chunks) # ['The quick brown fox', 'jumps over the', 'lazy dog.']Verify before relying
- Whether the claimed 15% RAG performance improvement over competitors is independently verified or from internal benchmarks.
- Current scale of 'millions of times per month' downloads and whether this refers to semchunk specifically or includes transitive installs.
- Specific performance characteristics (speed, memory usage) for large documents or high-concurrency scenarios.
Package facts
| License | MIT permissive |
| Python support | Supports the current Python release >=3.10 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 2 packagesdilltqdm |
| Maintenance | Actively maintained 62 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 3,936,290 / month, #2,444 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 5 - Production/StableIntended Audience :: DevelopersIntended Audience :: Information TechnologyIntended Audience :: Science/ResearchLicense :: OSI Approved :: MIT LicenseOperating System :: OS IndependentProgramming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Programming Language :: Python :: Implementation :: CPythonTopic :: Scientific/Engineering :: Artificial IntelligenceTopic :: Software Development :: Libraries :: Python ModulesTopic :: Text Processing :: GeneralTopic :: UtilitiesTyping :: Typed |
Evidence: semchunk-4.1.1-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “text chunking semantic context”
- semchunksemchunk splits text into semantically meaningful chunks while…
- chonkieChonkie splits text into semantically meaningful chunks for RAG…
- semantic-text-splitterSplits long text into semantically meaningful chunks sized for LLM…
Give your agent the search over MCP, or paste the wish link into any chat.
More Python Modules packages
Converts domain names between Unicode and ASCII-compatible encoding (Punycode) according to IDNA 2008 and Unicode Technical Standard 46, with security validation and broader script coverage than the standard library.
Install it if you work with internationalized domain names, need to validate domains, or use HTTP clients that depend on it transitively.
Setuptools is a Python build backend and package management tool that handles building, distributing, and installing Python packages, including support for C/C++ extension modules.
PyYAML parses and emits YAML 1.1 data format, enabling serialization and deserialization of configuration files and Python objects to and from human-readable YAML text.
Pydantic validates Python data structures against type hints, coercing and checking input at runtime to ensure it matches a declared schema.
Provides reusable metadata objects for use with PEP-593 `typing.Annotated` to express common constraints like bounds, collection sizes, and predicates on types.
Install it if you use or build libraries that need to express type constraints in a standardized, inspectable way—or if you want to annotate your own types with…
Provides runtime tools to inspect and introspect Python type annotations, enabling programmatic examination of type hints at execution time.
See also chonkie · memchunk · semantic-text-splitter · chonkie-core · tiktoken · aurelio-sdk · langchain-text-splitters · seltz · tensorflow-text · jieba3k