{"categories":[{"label":"Text Processing","url":"https://skillfed.io/packages/category/text-processing/3"}],"enrichment":{"capability":"Splits text into semantic chunks at delimiters (periods, newlines, custom patterns) using SIMD acceleration for high-throughput processing.","skillfed_tags":["simd-acceleration","rag-pipeline","zero-copy"],"use_cases":["Prepare large document collections for RAG systems by splitting text into semantic chunks before embedding.","Tokenize and chunk text for language model input pipelines where consistent chunk boundaries matter.","Process streaming or batch text data at high throughput when chunking is a bottleneck.","Split documents at custom delimiters (e.g., SentencePiece metaspace markers) for specialized NLP workflows.","Reduce memory overhead in text processing by using zero-copy memoryview chunks instead of string copies."],"what_it_does":"memchunk is a text chunking library optimized for speed using SIMD instructions and lookup tables. It splits text at semantic boundaries (periods, newlines, or custom delimiters) and returns chunks as zero-copy memoryview objects. The library is designed for high-throughput scenarios such as preparing documents for retrieval-augmented generation (RAG) pipelines or tokenization workflows.\n\nThe package provides a simple Chunker class that accepts text and configuration options: chunk size (default 4KB), delimiter characters, multi-byte patterns (useful for tokenizer-specific markers), and fallback strategies for finding split points. It supports Python 3.8 through 3.14 and is implemented in Rust with Python bindings, providing pre-compiled wheels for common platforms.","worth_installing":"Yes, if you need fast semantic text chunking for RAG, NLP pipelines, or high-volume document processing. The library is actively maintained, has no known vulnerabilities, supports modern Python versions, and offers a permissive dual license. Install friction is moderate due to compiled wheels, but pre-built binaries are available for all major platforms. Not necessary if chunking is not a performance bottleneck in your workflow."},"id":"memchunk","links":{"html":"https://skillfed.io/packages/memchunk","md":"https://skillfed.io/packages/memchunk.md","pypi":"https://pypi.org/project/memchunk/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-01-05","license_spdx":null,"license_treatment":"permissive","name":"memchunk","python_support":"supports_current","summary":"The fastest semantic text chunking library"},"popularity":{"monthly_downloads":96649,"position":13201,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.4.0"}