json-stream-rs-tokenizer

What it is and what it does

json-stream-rs-tokenizer is a Rust-compiled tokenizer for the json-stream library, ported from json-stream's own pure-Python tokenizer. Since json-stream 2.0, it is automatically used as a dependency, so explicit installation is typically unnecessary unless you're using an older json-stream version or need to ensure the Rust implementation is loaded. The package trades medium install friction (compiled wheels for multiple platforms, or Rust build from source) for parsing speedup.

The tokenizer has no runtime dependencies and works transparently: json-stream will use it if available and fall back to pure Python if the Rust extension failed to build or load. Explicit use requires importing RustTokenizer and passing it to json-stream's load or visit functions. PyPy users see only modest speedup, and large-integer conversion may fall back to Python in PyPy builds or when using Python's limited C API.

Use it for:

• Use with json-stream 2.0 or later to get automatic Rust-accelerated tokenization without code changes.
• Explicitly select RustTokenizer in older json-stream versions to override the default pure-Python implementation.
• Process large or deeply nested JSON documents where tokenization overhead is significant.
• Ensure consistent fast parsing across CPython environments without modifying application code.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

A Rust-based JSON tokenizer that accelerates JSON parsing for the json-stream library, available as a drop-in replacement or automatic dependency.

Yes, if you use json-stream 2.0 or later—it is installed automatically and provides transparent speedup with no code changes. Yes-with-conditions if using an older json-stream version: install it explicitly only if profiling shows tokenization is a bottleneck. No if you are on PyPy and the speedup does not justify the build complexity, or if your JSON workload is small and tokenization is not a measured bottleneck.

Install

json-stream-rs-tokenizer on PyPI

pip install json-stream-rs-tokenizer

uv add json-stream-rs-tokenizer

poetry add json-stream-rs-tokenizer

Installing json-stream-rs-tokenizer

Before you install

Medium install friction due to compiled Rust wheels; prebuilt wheels available for CPython 3.10–3.13 on macOS, Linux, and Windows. Installation falls back gracefully if compilation fails, so the package can be a dependency without blocking installation.

License in practice

Project code is MIT-licensed, but compiled wheel binaries include Rust dependencies under different permissive licenses. Refer to LICENSES-RUST-DEPS.html and LICENSES-RUST.html files before use in proprietary or restricted contexts.

Quickstart

pip install json-stream-rs-tokenizer

from json_stream_rs_tokenizer import RustTokenizer
from json_stream import load
from io import StringIO

json_buf = StringIO('{"a": [1,2,3,4], "b": [5,6,7]}')
d = load(json_buf, tokenizer=RustTokenizer)

Requires Rust toolchain if prebuilt wheel unavailable for your platform; build from source may fail silently, leaving RustTokenizer unavailable for import.

Verify before relying

• Whether the 4–10x speedup factor applies to your specific JSON data shape and size.
• Whether PyPy speedup of 1.0–1.5x is acceptable for your workload.
• Whether fallback to pure-Python tokenizer when Rust compilation fails meets your performance requirements.

Package facts

Evidence: json_stream_rs_tokenizer-0.5.3-cp310-cp310-macosx_10_12_x86_64.whl; json_stream_rs_tokenizer-0.5.3-cp310-cp310-macosx_11_0_arm64.whl; json_stream_rs_tokenizer-0.5.3-cp310-cp310-manylinux_2_28_aarch64.whl; json_stream_rs_tokenizer-0.5.3-cp310-cp310-manylinux_2_28_x86_64.whl; json_stream_rs_tokenizer-0.5.3-cp310-cp310-win_amd64.whl; json_stream_rs_tokenizer-0.5.3-cp311-cp311-macosx_10_12_x86_64.whl; json_stream_rs_tokenizer-0.5.3-cp311-cp311-macosx_11_0_arm64.whl; json_stream_rs_tokenizer-0.5.3-cp311-cp311-manylinux_2_28_aarch64.whl; json_stream_rs_tokenizer-0.5.3-cp311-cp311-manylinux_2_28_x86_64.whl; json_stream_rs_tokenizer-0.5.3-cp311-cp311-win_amd64.whl; json_stream_rs_tokenizer-0.5.3-cp312-cp312-macosx_10_13_x86_64.whl; json_stream_rs_tokenizer-0.5.3-cp312-cp312-macosx_11_0_arm64.whl; json_stream_rs_tokenizer-0.5.3-cp312-cp312-manylinux_2_28_aarch64.whl; json_stream_rs_tokenizer-0.5.3-cp312-cp312-manylinux_2_28_x86_64.whl; json_stream_rs_tokenizer-0.5.3-cp312-cp312-win_amd64.whl; json_stream_rs_tokenizer-0.5.3-cp313-cp313-macosx_10_13_x86_64.whl; json_stream_rs_tokenizer-0.5.3-cp313-cp313-macosx_11_0_arm64.whl; json_stream_rs_tokenizer-0.5.3-cp313-cp313-manylinux_2_28_aarch64.whl; json_stream_rs_tokenizer-0.5.3-cp313-cp313-manylinux_2_28_x86_64.whl; json_stream_rs_tokenizer-0.5.3-cp313-cp313-win_amd64.whl

Tags

More Text Processing packages