chonkie

What it is and what it does

Chonkie is a text chunking library designed to prepare documents for retrieval-augmented generation (RAG) systems. It provides multiple chunking strategies—including recursive, semantic, token-based, code-aware, and neural approaches—each suited to different content types and use cases. The library integrates with tokenizers, embedding models, vector databases, and LLMs, allowing you to build end-to-end pipelines that fetch, chunk, refine, embed, and load data into your RAG infrastructure.

The package emphasizes minimal dependencies by default: the base install includes only what's needed for common chunking tasks, with optional extras for specialized features like semantic chunking, code analysis, or API server deployment. It supports both synchronous and asynchronous workflows, can run as a self-hosted REST API, and handles text preprocessing through pluggable "Chef" components for markdown, tables, and OCR.

Use it for:

• Split long documents into token-bounded chunks for LLM context windows before embedding and retrieval
• Chunk code repositories by syntactic structure to preserve function and class boundaries for code search
• Build a RAG pipeline that chains recursive chunking, semantic refinement, and embedding in a single workflow
• Run Chonkie as a microservice API to chunk documents from multiple applications without duplicating logic
• Process markdown or CSV files into structured chunks with context overlap for better retrieval quality

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Chonkie splits text into semantically meaningful chunks for RAG pipelines, offering multiple chunking strategies (recursive, semantic, token-based, code-aware) plus refinement and embedding integration.

Yes. Chonkie is actively maintained, has no known vulnerabilities, uses a permissive MIT license, and offers low install friction with a focused set of dependencies. It solves a genuine problem in RAG pipelines—text chunking—with multiple strategies and integrations. The modular design lets you install only what you need, and the API server option adds deployment flexibility. Suitable for production RAG systems.

Install

chonkie on PyPI

pip install chonkie

uv add chonkie

poetry add chonkie

Installing chonkie

Before you install

Low friction install with a pure-Python wheel and six runtime dependencies. Active maintenance with recent commits and 4673 repository stars. Supports Python 3.10 through 3.13.

License in practice

MIT License permits unrestricted use, modification, and redistribution in commercial and private projects with minimal obligations—only attribution and license inclusion required.

Quickstart

pip install chonkie

from chonkie import RecursiveChunker

chunker = RecursiveChunker()
chunks = chunker("Your text here")
for chunk in chunks:
    print(chunk.text)

Requires Python 3.10 or later.

Verify before relying

• Whether all 45+ integrations mentioned in the description are available in the base install or require optional extras
• Performance characteristics of different chunkers (e.g., the claimed '100+ GB/s' for FastChunker)
• Multilingual support coverage across the stated 56 languages

Package facts

Evidence: chonkie-1.7.0-py3-none-any.whl

Keywords: chunking, rag, retrieval-augmented-generation, nlp, natural-language-processing, text-processing, text-analysis, text-chunking, artificial-intelligence, machine-learning

Tags

More Artificial Intelligence packages