{"categories":[{"label":"Text Processing","url":"https://skillfed.io/packages/category/text-processing/2"}],"enrichment":{"capability":"chonkie-core splits text at semantic boundaries (periods, newlines, custom delimiters) into fixed-size chunks, returning zero-copy memoryview objects with configurable size and splitting behavior.","skillfed_tags":["text-chunking","rag-pipeline","rust-extension"],"use_cases":["Prepare large documents for RAG systems by splitting text into semantic chunks before embedding.","Tokenize and segment text for NLP pipelines that require fixed-size input windows.","Split multi-byte patterns (e.g., SentencePiece metaspace markers) for specialized tokenizers.","Batch-process large text corpora with minimal memory overhead using zero-copy memoryview slices.","Handle consecutive delimiters (e.g., multiple spaces) by splitting at run boundaries rather than within runs."],"what_it_does":"chonkie-core is a text chunking library implemented in Rust and exposed as a Python extension. It splits input text into fixed-size chunks at semantic boundaries\u2014periods, newlines, or custom delimiters you specify\u2014and returns the results as memoryview objects (zero-copy slices of the original text). The library supports configurable chunk size, custom delimiter sets, multi-byte patterns (useful for tokenizer-specific markers), and fallback strategies for cases where no delimiter appears in the backward search window.\n\nThe package is designed for high-throughput text processing, particularly in RAG (retrieval-augmented generation) pipelines and NLP workflows where you need to prepare large documents for embedding or indexing. It has no runtime dependencies beyond Python itself, ships with prebuilt wheels for modern Python versions (3.10\u20133.14) across macOS, Linux, and Windows, and is actively maintained.","worth_installing":"Yes. chonkie-core is actively maintained, has no dependencies, installs cleanly on modern Python (3.10+) across all major platforms, carries permissive dual licensing, and solves a real problem\u2014fast semantic text chunking\u2014with a straightforward API. The lack of security vulnerabilities and recent activity (last commit May 2026) add confidence. Install it if you need to chunk text for RAG or NLP workflows."},"id":"chonkie-core","links":{"html":"https://skillfed.io/packages/chonkie-core","md":"https://skillfed.io/packages/chonkie-core.md","pypi":"https://pypi.org/project/chonkie-core/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-05-28","license_spdx":null,"license_treatment":"permissive","name":"chonkie-core","python_support":"supports_current","summary":"The fastest semantic text chunking library"},"popularity":{"monthly_downloads":644236,"position":5602,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.10.2"}