{"categories":[{"label":"Python Modules","url":"https://skillfed.io/packages/category/software-development-libraries-python-modules/5"},{"label":"Utilities","url":"https://skillfed.io/packages/category/utilities/2"},{"label":"Artificial Intelligence","url":"https://skillfed.io/packages/category/scientific-engineering-artificial-intelligence/2"},{"label":"General","url":"https://skillfed.io/packages/category/text-processing-general"}],"enrichment":{"capability":"semchunk splits text into semantically meaningful chunks while preserving local context, supporting custom tokenizers, chunk overlapping, offsets, and optional AI-powered chunking via the Isaacus API.","skillfed_tags":["rag-preprocessing","text-splitting","semantic-chunking"],"use_cases":["Prepare documents for RAG pipelines by splitting them into token-bounded chunks that preserve semantic coherence for embedding and retrieval.","Batch-process large text corpora with multiprocessing, tracking chunk offsets to reconstruct source positions after retrieval.","Integrate custom tokenizers (e.g., domain-specific or model-specific) into a chunking workflow without rewriting splitting logic.","Overlap chunks for sliding-window context in language model fine-tuning or evaluation tasks.","Use AI-powered chunking (via Isaacus) to make semantic split decisions for complex documents like legal or scientific texts."],"what_it_does":"semchunk is a Python library that breaks text into smaller, semantically coherent chunks\u2014a critical preprocessing step for retrieval-augmented generation (RAG) and other NLP workflows. It uses a hierarchical chunking algorithm to preserve local semantic context better than simpler splitting strategies. The library is agnostic about tokenization: you can supply any tokenizer (Tiktoken, Hugging Face Transformers, or a custom function) or a simple token counter, and semchunk will respect your token budget while keeping related text together.\n\nThe library supports chunk overlapping (by ratio or absolute token count), offset tracking (to map chunks back to source positions), and multiprocessing for batch operations. Optionally, you can enable AI-powered chunking by providing an Isaacus enrichment model name and API key, which uses semantic understanding to make smarter split decisions. It requires only dill and tqdm as dependencies, installs as a pure Python wheel, and supports Python 3.10 through 3.14.","worth_installing":"Yes. semchunk is production-ready, actively maintained, has no known vulnerabilities, and solves a real problem (semantic text chunking) with low install friction and a permissive license. It is widely used (top 5000 PyPI packages) and offers flexibility via custom tokenizers and optional AI enhancement. Install it if you need to chunk text for RAG, embeddings, or language model workflows."},"id":"semchunk","links":{"html":"https://skillfed.io/packages/semchunk","md":"https://skillfed.io/packages/semchunk.md","pypi":"https://pypi.org/project/semchunk/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-06-13","license_spdx":null,"license_treatment":"permissive","name":"semchunk","python_support":"supports_current","summary":"A Python library for splitting text into smaller chunks while preserving as much local semantic context as possible."},"popularity":{"monthly_downloads":3936290,"position":2444,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"4.1.1"}