---
id: memchunk
version: "0.4.0"
license: MIT OR Apache-2.0
license_treatment: permissive
maintenance: active
---

# memchunk — The fastest semantic text chunking library

License: permissive · Maintenance: active · Downloads: 96.6K/mo

## What it is and what it does

memchunk is a text chunking library optimized for speed using SIMD instructions and lookup tables. It splits text at semantic boundaries (periods, newlines, or custom delimiters) and returns chunks as zero-copy memoryview objects. The library is designed for high-throughput scenarios such as preparing documents for retrieval-augmented generation (RAG) pipelines or tokenization workflows.

The package provides a simple Chunker class that accepts text and configuration options: chunk size (default 4KB), delimiter characters, multi-byte patterns (useful for tokenizer-specific markers), and fallback strategies for finding split points. It supports Python 3.8 through 3.14 and is implemented in Rust with Python bindings, providing pre-compiled wheels for common platforms.

Use it for:

- Prepare large document collections for RAG systems by splitting text into semantic chunks before embedding.
- Tokenize and chunk text for language model input pipelines where consistent chunk boundaries matter.
- Process streaming or batch text data at high throughput when chunking is a bottleneck.
- Split documents at custom delimiters (e.g., SentencePiece metaspace markers) for specialized NLP workflows.
- Reduce memory overhead in text processing by using zero-copy memoryview chunks instead of string copies.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Splits text into semantic chunks at delimiters (periods, newlines, custom patterns) using SIMD acceleration for high-throughput processing.

Yes, if you need fast semantic text chunking for RAG, NLP pipelines, or high-volume document processing. The library is actively maintained, has no known vulnerabilities, supports modern Python versions, and offers a permissive dual license. Install friction is moderate due to compiled wheels, but pre-built binaries are available for all major platforms. Not necessary if chunking is not a performance bottleneck in your workflow.

## Install

pip install memchunk
uv add memchunk
poetry add memchunk

## Installing memchunk

Before you install: Medium install friction due to compiled wheels, but pre-built binaries are available for Python 3.8–3.14 across Linux, macOS (x86_64 and ARM), and Windows. Active maintenance with recent releases.

License in practice: Dual-licensed under MIT or Apache-2.0 (permissive); you may choose either license. No restrictions on commercial or proprietary use.

Quickstart:

from memchunk import Chunker

text = "Hello world. How are you?"
for chunk in Chunker(text):
    print(bytes(chunk))

# Custom size and delimiters
for chunk in Chunker(text, size=1024, delimiters=".?!\n"):
    print(bytes(chunk))

Requires Python 3.8 or later. Pre-compiled wheels are provided; source builds require a Rust toolchain.

Verify before relying:

- Actual throughput benchmarks and whether '1 TB/s' claim is measured on specific hardware or representative of typical use.
- Memory overhead of the Chunker object and whether it scales linearly with text size.
- Behavior when text contains multi-byte UTF-8 sequences at chunk boundaries.

## Package facts

- License: MIT OR Apache-2.0 (permissive)
- Python support: supports_current
- Install friction: medium
- Maintenance: active
- Downloads: 96.6K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

text chunking library, semantic text splitting, fast document chunking, simd text processing, rag text preparation, tokenization chunking, high-throughput text splitting, simd-acceleration, rag-pipeline, zero-copy

[View on SkillFed](https://skillfed.io/packages/memchunk) · [View on PyPI](https://pypi.org/project/memchunk/)