memchunk

What it is and what it does

memchunk is a text chunking library optimized for speed using SIMD instructions and lookup tables. It splits text at semantic boundaries (periods, newlines, or custom delimiters) and returns chunks as zero-copy memoryview objects. The library is designed for high-throughput scenarios such as preparing documents for retrieval-augmented generation (RAG) pipelines or tokenization workflows.

The package provides a simple Chunker class that accepts text and configuration options: chunk size (default 4KB), delimiter characters, multi-byte patterns (useful for tokenizer-specific markers), and fallback strategies for finding split points. It supports Python 3.8 through 3.14 and is implemented in Rust with Python bindings, providing pre-compiled wheels for common platforms.

Use it for:

• Prepare large document collections for RAG systems by splitting text into semantic chunks before embedding.
• Tokenize and chunk text for language model input pipelines where consistent chunk boundaries matter.
• Process streaming or batch text data at high throughput when chunking is a bottleneck.
• Split documents at custom delimiters (e.g., SentencePiece metaspace markers) for specialized NLP workflows.
• Reduce memory overhead in text processing by using zero-copy memoryview chunks instead of string copies.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Splits text into semantic chunks at delimiters (periods, newlines, custom patterns) using SIMD acceleration for high-throughput processing.

Yes, if you need fast semantic text chunking for RAG, NLP pipelines, or high-volume document processing. The library is actively maintained, has no known vulnerabilities, supports modern Python versions, and offers a permissive dual license. Install friction is moderate due to compiled wheels, but pre-built binaries are available for all major platforms. Not necessary if chunking is not a performance bottleneck in your workflow.

Install

memchunk on PyPI

pip install memchunk

uv add memchunk

poetry add memchunk

Installing memchunk

Before you install

Medium install friction due to compiled wheels, but pre-built binaries are available for Python 3.8–3.14 across Linux, macOS (x86_64 and ARM), and Windows. Active maintenance with recent releases.

License in practice

Dual-licensed under MIT or Apache-2.0 (permissive); you may choose either license. No restrictions on commercial or proprietary use.

Quickstart

from memchunk import Chunker

text = "Hello world. How are you?"
for chunk in Chunker(text):
    print(bytes(chunk))

# Custom size and delimiters
for chunk in Chunker(text, size=1024, delimiters=".?!\n"):
    print(bytes(chunk))

Requires Python 3.8 or later. Pre-compiled wheels are provided; source builds require a Rust toolchain.

Verify before relying

• Actual throughput benchmarks and whether '1 TB/s' claim is measured on specific hardware or representative of typical use.
• Memory overhead of the Chunker object and whether it scales linearly with text size.
• Behavior when text contains multi-byte UTF-8 sequences at chunk boundaries.

Package facts

Evidence: memchunk-0.4.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; memchunk-0.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; memchunk-0.4.0-cp310-cp310-win_amd64.whl; memchunk-0.4.0-cp311-cp311-macosx_10_12_x86_64.whl; memchunk-0.4.0-cp311-cp311-macosx_11_0_arm64.whl; memchunk-0.4.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; memchunk-0.4.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; memchunk-0.4.0-cp311-cp311-win_amd64.whl; memchunk-0.4.0-cp312-cp312-macosx_10_12_x86_64.whl; memchunk-0.4.0-cp312-cp312-macosx_11_0_arm64.whl; memchunk-0.4.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; memchunk-0.4.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; memchunk-0.4.0-cp312-cp312-win_amd64.whl; memchunk-0.4.0-cp313-cp313-macosx_10_12_x86_64.whl; memchunk-0.4.0-cp313-cp313-macosx_11_0_arm64.whl; memchunk-0.4.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; memchunk-0.4.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; memchunk-0.4.0-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; memchunk-0.4.0-cp313-cp313-win_amd64.whl; memchunk-0.4.0-cp314-cp314-macosx_10_12_x86_64.whl

Keywords: chunking, text, simd, nlp, tokenization, rag

Tags

More Text Processing packages