chonkie-core

What it is and what it does

chonkie-core is a text chunking library implemented in Rust and exposed as a Python extension. It splits input text into fixed-size chunks at semantic boundaries—periods, newlines, or custom delimiters you specify—and returns the results as memoryview objects (zero-copy slices of the original text). The library supports configurable chunk size, custom delimiter sets, multi-byte patterns (useful for tokenizer-specific markers), and fallback strategies for cases where no delimiter appears in the backward search window.

The package is designed for high-throughput text processing, particularly in RAG (retrieval-augmented generation) pipelines and NLP workflows where you need to prepare large documents for embedding or indexing. It has no runtime dependencies beyond Python itself, ships with prebuilt wheels for modern Python versions (3.10–3.14) across macOS, Linux, and Windows, and is actively maintained.

Use it for:

• Prepare large documents for RAG systems by splitting text into semantic chunks before embedding.
• Tokenize and segment text for NLP pipelines that require fixed-size input windows.
• Split multi-byte patterns (e.g., SentencePiece metaspace markers) for specialized tokenizers.
• Batch-process large text corpora with minimal memory overhead using zero-copy memoryview slices.
• Handle consecutive delimiters (e.g., multiple spaces) by splitting at run boundaries rather than within runs.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

chonkie-core splits text at semantic boundaries (periods, newlines, custom delimiters) into fixed-size chunks, returning zero-copy memoryview objects with configurable size and splitting behavior.

Yes. chonkie-core is actively maintained, has no dependencies, installs cleanly on modern Python (3.10+) across all major platforms, carries permissive dual licensing, and solves a real problem—fast semantic text chunking—with a straightforward API. The lack of security vulnerabilities and recent activity (last commit May 2026) add confidence. Install it if you need to chunk text for RAG or NLP workflows.

Install

chonkie-core on PyPI

pip install chonkie-core

uv add chonkie-core

poetry add chonkie-core

Installing chonkie-core

Before you install

Medium friction: compiled Rust extension with prebuilt wheels for Python 3.10–3.14 on macOS (x86_64, arm64), Linux (x86_64, aarch64), and Windows (amd64). Active maintenance (last commit 2026-05-28, 78 days since release); no runtime dependencies.

License in practice

Dual-licensed under MIT or Apache-2.0 at your option—both permissive, so you can choose whichever fits your project's license strategy with no restrictions on commercial or private use.

Quickstart

from chonkie_core import Chunker

text = "Hello world. How are you?"
for chunk in Chunker(text, size=1024, delimiters=".?!\n"):
    print(bytes(chunk))

Requires Python ≥3.10; prebuilt wheels available for common platforms, but installation may require a compatible wheel for your architecture.

Verify before relying

• Actual throughput claims (e.g., '1 TB/s', 'Wikipedia in 120ms') are not independently verified in the fact sheet.
• Performance comparison to other chunking libraries is not documented in the fact sheet.
• Whether memoryview zero-copy behavior is preserved across all chunking modes is not explicitly confirmed.

Package facts

Evidence: chonkie_core-0.10.2-cp310-cp310-macosx_10_12_x86_64.whl; chonkie_core-0.10.2-cp310-cp310-macosx_11_0_arm64.whl; chonkie_core-0.10.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; chonkie_core-0.10.2-cp310-cp310-manylinux_2_28_aarch64.whl; chonkie_core-0.10.2-cp310-cp310-win_amd64.whl; chonkie_core-0.10.2-cp311-cp311-macosx_10_12_x86_64.whl; chonkie_core-0.10.2-cp311-cp311-macosx_11_0_arm64.whl; chonkie_core-0.10.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; chonkie_core-0.10.2-cp311-cp311-manylinux_2_28_aarch64.whl; chonkie_core-0.10.2-cp311-cp311-win_amd64.whl; chonkie_core-0.10.2-cp312-cp312-macosx_10_12_x86_64.whl; chonkie_core-0.10.2-cp312-cp312-macosx_11_0_arm64.whl; chonkie_core-0.10.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; chonkie_core-0.10.2-cp312-cp312-manylinux_2_28_aarch64.whl; chonkie_core-0.10.2-cp312-cp312-win_amd64.whl; chonkie_core-0.10.2-cp313-cp313-macosx_10_12_x86_64.whl; chonkie_core-0.10.2-cp313-cp313-macosx_11_0_arm64.whl; chonkie_core-0.10.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; chonkie_core-0.10.2-cp313-cp313-manylinux_2_28_aarch64.whl; chonkie_core-0.10.2-cp313-cp313-win_amd64.whl

Keywords: chunking, text, simd, nlp, tokenization, rag, chonkie

Tags

More Text Processing packages