chonkie-core
The fastest semantic text chunking library
Decision gist · record as of 2026-08-14
Yes. chonkie-core is actively maintained, has no dependencies, installs cleanly on modern Python (3.10+) across all major platforms, carries permissive dual licensing, and solves a real problem—fast semantic text chunking—with a straightforward API. The lack of security vulnerabilities and recent activity (last commit May 2026) add confidence. Install it if you need to chunk text for RAG or NLP workflows.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python ≥3.10; prebuilt wheels available for common platforms, but installation may require a compatible wheel for your architecture.
- Medium friction: compiled Rust extension with prebuilt wheels for Python 3.10–3.14 on macOS (x86_64, arm64), Linux (x86_64, aarch64), and Windows (amd64).
- Active maintenance (last commit 2026-05-28, 78 days since release); no runtime dependencies.
License · maintenance · safety
MIT OR Apache-2.0 (permissive) — Dual-licensed under MIT or Apache-2.0 at your option—both permissive, so you can choose whichever fits your project's license strategy with no restrictions on commercial or private use.
last release 2026-05-28 (78 days) · last repo commit 2026-05-28 · 358 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 644,236 downloads/mo, #5,602 on PyPI
Alternatives
Verify before relying
from chonkie_core import Chunker
text = "Hello world. How are you?"
for chunk in Chunker(text, size=1024, delimiters=".?!\n"):
print(bytes(chunk))- Actual throughput claims (e.g., '1 TB/s', 'Wikipedia in 120ms') are not independently verified in the fact sheet.
- Performance comparison to other chunking libraries is not documented in the fact sheet.
- Whether memoryview zero-copy behavior is preserved across all chunking modes is not explicitly confirmed.
What it is and what it does
chonkie-core is a text chunking library implemented in Rust and exposed as a Python extension. It splits input text into fixed-size chunks at semantic boundaries—periods, newlines, or custom delimiters you specify—and returns the results as memoryview objects (zero-copy slices of the original text). The library supports configurable chunk size, custom delimiter sets, multi-byte patterns (useful for tokenizer-specific markers), and fallback strategies for cases where no delimiter appears in the backward search window.
The package is designed for high-throughput text processing, particularly in RAG (retrieval-augmented generation) pipelines and NLP workflows where you need to prepare large documents for embedding or indexing. It has no runtime dependencies beyond Python itself, ships with prebuilt wheels for modern Python versions (3.10–3.14) across macOS, Linux, and Windows, and is actively maintained.
Use it for
- Prepare large documents for RAG systems by splitting text into semantic chunks before embedding.
- Tokenize and segment text for NLP pipelines that require fixed-size input windows.
- Split multi-byte patterns (e.g., SentencePiece metaspace markers) for specialized tokenizers.
- Batch-process large text corpora with minimal memory overhead using zero-copy memoryview slices.
- Handle consecutive delimiters (e.g., multiple spaces) by splitting at run boundaries rather than within runs.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
chonkie-core is actively maintained, has no dependencies, installs cleanly on modern Python (3.10+) across all major platforms, carries permissive dual licensing, and solves a real problem—fast semantic text chunking—with a straightforward API. The lack of security vulnerabilities and recent activity (last commit May 2026) add confidence. Install it if you need to chunk text for RAG or NLP workflows.
Install
chonkie-core on PyPI
Before you install
Medium friction: compiled Rust extension with prebuilt wheels for Python 3.10–3.14 on macOS (x86_64, arm64), Linux (x86_64, aarch64), and Windows (amd64). Active maintenance (last commit 2026-05-28, 78 days since release); no runtime dependencies.
Requires Python ≥3.10; prebuilt wheels available for common platforms, but installation may require a compatible wheel for your architecture.
License in practice
Dual-licensed under MIT or Apache-2.0 at your option—both permissive, so you can choose whichever fits your project's license strategy with no restrictions on commercial or private use.
Quickstart
from chonkie_core import Chunker
text = "Hello world. How are you?"
for chunk in Chunker(text, size=1024, delimiters=".?!\n"):
print(bytes(chunk))Verify before relying
- Actual throughput claims (e.g., '1 TB/s', 'Wikipedia in 120ms') are not independently verified in the fact sheet.
- Performance comparison to other chunking libraries is not documented in the fact sheet.
- Whether memoryview zero-copy behavior is preserved across all chunking modes is not explicitly confirmed.
Package facts
| License | MIT OR Apache-2.0 permissive |
| Python support | Supports the current Python release >=3.10 |
| Install friction | Medium. Platform-specific wheel |
| Runtime dependencies | None |
| Maintenance | Actively maintained 78 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 644,236 / month, #5,602 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 4 - BetaIntended Audience :: DevelopersLicense :: OSI Approved :: Apache Software LicenseLicense :: OSI Approved :: MIT LicenseProgramming Language :: Python :: 3Programming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Programming Language :: Python :: Implementation :: CPythonProgramming Language :: RustTopic :: Text Processing |
Evidence: chonkie_core-0.10.2-cp310-cp310-macosx_10_12_x86_64.whl; chonkie_core-0.10.2-cp310-cp310-macosx_11_0_arm64.whl; chonkie_core-0.10.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; chonkie_core-0.10.2-cp310-cp310-manylinux_2_28_aarch64.whl; chonkie_core-0.10.2-cp310-cp310-win_amd64.whl; chonkie_core-0.10.2-cp311-cp311-macosx_10_12_x86_64.whl; chonkie_core-0.10.2-cp311-cp311-macosx_11_0_arm64.whl; chonkie_core-0.10.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; chonkie_core-0.10.2-cp311-cp311-manylinux_2_28_aarch64.whl; chonkie_core-0.10.2-cp311-cp311-win_amd64.whl; chonkie_core-0.10.2-cp312-cp312-macosx_10_12_x86_64.whl; chonkie_core-0.10.2-cp312-cp312-macosx_11_0_arm64.whl; chonkie_core-0.10.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; chonkie_core-0.10.2-cp312-cp312-manylinux_2_28_aarch64.whl; chonkie_core-0.10.2-cp312-cp312-win_amd64.whl; chonkie_core-0.10.2-cp313-cp313-macosx_10_12_x86_64.whl; chonkie_core-0.10.2-cp313-cp313-macosx_11_0_arm64.whl; chonkie_core-0.10.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; chonkie_core-0.10.2-cp313-cp313-manylinux_2_28_aarch64.whl; chonkie_core-0.10.2-cp313-cp313-win_amd64.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “chunk text by delimiters”
- chonkie-corechonkie-core splits text at semantic boundaries (periods, newlines,…
- memchunkSplits text into semantic chunks at delimiters (periods, newlines,…
- more-itertoolsProvides additional iterator utilities and recipes that extend…
Give your agent the search over MCP, or paste the wish link into any chat.
More Text Processing packages
A drop-in replacement for Python's standard `re` module that adds advanced regex features like nested sets, fuzzy matching, lookaround in conditionals, and full Unicode case-folding while maintaining backward compatibility.
pyparsing provides a library for building text parsers directly in Python code using composable grammar classes, handling quoted strings, whitespace variation, and embedded comments without regex or lex/yacc.
Install it if you need to parse text or define grammars programmatically.
fonttools manipulates font files in multiple formats (TrueType, OpenType, AFM, Type 1, Mac-specific) and includes TTX, a tool to convert fonts to and from XML text format.
Install it if you need to read, write, or manipulate fonts programmatically or via the TTX command-line tool.
Docutils converts plaintext documentation in reStructuredText format into multiple output formats including HTML, XML, and LaTeX using a modular processing system.
RapidFuzz provides fast fuzzy string matching using Levenshtein Distance and related metrics, implemented mostly in C++ with Python bindings for rapid similarity scoring and approximate string matching.
Install it if you need fuzzy string matching; it's a solid replacement for FuzzyWuzzy with better licensing and performance.
tinycss2 parses CSS strings into token and block objects, and generates CSS strings from those objects, following the CSS Syntax Level 3 specification without enforcing specific properties or values.
Install it if your project requires CSS tokenization or syntax manipulation.
See also chonkie · memchunk · semantic-text-splitter · semchunk · langchain-text-splitters · sentence-stream · icechunk · curated-tokenizers · jieba3k · litdata