json-stream-rs-tokenizer
A faster tokenizer for the json-stream Python library
Decision gist · record as of 2026-08-14
Yes, if you use json-stream 2.0 or later—it is installed automatically and provides transparent speedup with no code changes. Yes-with-conditions if using an older json-stream version: install it explicitly only if profiling shows tokenization is a bottleneck. No if you are on PyPy and the speedup does not justify the build complexity, or if your JSON workload is small and tokenization is not a measured bottleneck.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Rust toolchain if prebuilt wheel unavailable for your platform; build from source may fail silently, leaving RustTokenizer unavailable for import.
- Medium install friction due to compiled Rust wheels; prebuilt wheels available for CPython 3.10–3.13 on macOS, Linux, and Windows.
- Installation falls back gracefully if compilation fails, so the package can be a dependency without blocking installation.
License · maintenance · safety
(unclear) — Project code is MIT-licensed, but compiled wheel binaries include Rust dependencies under different permissive licenses. Refer to LICENSES-RUST-DEPS.html and LICENSES-RUST.html files before use in proprietary or restricted contexts.
last release 2026-08-10 (4 days) · last repo commit 2026-08-10 · 10 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 1,253,425 downloads/mo, #4,158 on PyPI
Alternatives
Verify before relying
pip install json-stream-rs-tokenizer
from json_stream_rs_tokenizer import RustTokenizer
from json_stream import load
from io import StringIO
json_buf = StringIO('{"a": [1,2,3,4], "b": [5,6,7]}')
d = load(json_buf, tokenizer=RustTokenizer)- Whether the 4–10x speedup factor applies to your specific JSON data shape and size.
- Whether PyPy speedup of 1.0–1.5x is acceptable for your workload.
- Whether fallback to pure-Python tokenizer when Rust compilation fails meets your performance requirements.
What it is and what it does
json-stream-rs-tokenizer is a Rust-compiled tokenizer for the json-stream library, ported from json-stream's own pure-Python tokenizer. Since json-stream 2.0, it is automatically used as a dependency, so explicit installation is typically unnecessary unless you're using an older json-stream version or need to ensure the Rust implementation is loaded. The package trades medium install friction (compiled wheels for multiple platforms, or Rust build from source) for parsing speedup.
The tokenizer has no runtime dependencies and works transparently: json-stream will use it if available and fall back to pure Python if the Rust extension failed to build or load. Explicit use requires importing RustTokenizer and passing it to json-stream's load or visit functions. PyPy users see only modest speedup, and large-integer conversion may fall back to Python in PyPy builds or when using Python's limited C API.
Use it for
- Use with json-stream 2.0 or later to get automatic Rust-accelerated tokenization without code changes.
- Explicitly select RustTokenizer in older json-stream versions to override the default pure-Python implementation.
- Process large or deeply nested JSON documents where tokenization overhead is significant.
- Ensure consistent fast parsing across CPython environments without modifying application code.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you use json-stream 2.0 or later—it is installed automatically and provides transparent speedup with no code changes.
Yes-with-conditions if using an older json-stream version: install it explicitly only if profiling shows tokenization is a bottleneck. No if you are on PyPy and the speedup does not justify the build complexity, or if your JSON workload is small and tokenization is not a measured bottleneck.
Install
json-stream-rs-tokenizer on PyPI
Before you install
Medium install friction due to compiled Rust wheels; prebuilt wheels available for CPython 3.10–3.13 on macOS, Linux, and Windows. Installation falls back gracefully if compilation fails, so the package can be a dependency without blocking installation.
Requires Rust toolchain if prebuilt wheel unavailable for your platform; build from source may fail silently, leaving RustTokenizer unavailable for import.
License in practice
Project code is MIT-licensed, but compiled wheel binaries include Rust dependencies under different permissive licenses. Refer to LICENSES-RUST-DEPS.html and LICENSES-RUST.html files before use in proprietary or restricted contexts.
Quickstart
pip install json-stream-rs-tokenizer
from json_stream_rs_tokenizer import RustTokenizer
from json_stream import load
from io import StringIO
json_buf = StringIO('{"a": [1,2,3,4], "b": [5,6,7]}')
d = load(json_buf, tokenizer=RustTokenizer)Verify before relying
- Whether the 4–10x speedup factor applies to your specific JSON data shape and size.
- Whether PyPy speedup of 1.0–1.5x is acceptable for your workload.
- Whether fallback to pure-Python tokenizer when Rust compilation fails meets your performance requirements.
Package facts
| License | Not declared unclear |
| Python support | Supports the current Python release <3.15,>=3.8 |
| Install friction | Medium. Platform-specific wheel |
| Runtime dependencies | None |
| Maintenance | Actively maintained 4 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 1,253,425 / month, #4,158 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Programming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Programming Language :: Python :: 3.8Programming Language :: Python :: 3.9Programming Language :: Python :: Implementation :: CPythonProgramming Language :: Python :: Implementation :: PyPyProgramming Language :: Rust |
Evidence: json_stream_rs_tokenizer-0.5.3-cp310-cp310-macosx_10_12_x86_64.whl; json_stream_rs_tokenizer-0.5.3-cp310-cp310-macosx_11_0_arm64.whl; json_stream_rs_tokenizer-0.5.3-cp310-cp310-manylinux_2_28_aarch64.whl; json_stream_rs_tokenizer-0.5.3-cp310-cp310-manylinux_2_28_x86_64.whl; json_stream_rs_tokenizer-0.5.3-cp310-cp310-win_amd64.whl; json_stream_rs_tokenizer-0.5.3-cp311-cp311-macosx_10_12_x86_64.whl; json_stream_rs_tokenizer-0.5.3-cp311-cp311-macosx_11_0_arm64.whl; json_stream_rs_tokenizer-0.5.3-cp311-cp311-manylinux_2_28_aarch64.whl; json_stream_rs_tokenizer-0.5.3-cp311-cp311-manylinux_2_28_x86_64.whl; json_stream_rs_tokenizer-0.5.3-cp311-cp311-win_amd64.whl; json_stream_rs_tokenizer-0.5.3-cp312-cp312-macosx_10_13_x86_64.whl; json_stream_rs_tokenizer-0.5.3-cp312-cp312-macosx_11_0_arm64.whl; json_stream_rs_tokenizer-0.5.3-cp312-cp312-manylinux_2_28_aarch64.whl; json_stream_rs_tokenizer-0.5.3-cp312-cp312-manylinux_2_28_x86_64.whl; json_stream_rs_tokenizer-0.5.3-cp312-cp312-win_amd64.whl; json_stream_rs_tokenizer-0.5.3-cp313-cp313-macosx_10_13_x86_64.whl; json_stream_rs_tokenizer-0.5.3-cp313-cp313-macosx_11_0_arm64.whl; json_stream_rs_tokenizer-0.5.3-cp313-cp313-manylinux_2_28_aarch64.whl; json_stream_rs_tokenizer-0.5.3-cp313-cp313-manylinux_2_28_x86_64.whl; json_stream_rs_tokenizer-0.5.3-cp313-cp313-win_amd64.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “fast json tokenizer”
- json-stream-rs-tokenizerA Rust-based JSON tokenizer that accelerates JSON parsing for the…
- tokieA fast, Rust-backed tokenizer library that encodes and decodes text…
- textparserA fast text parser library that lets you define grammars using token…
Give your agent the search over MCP, or paste the wish link into any chat.
More Text Processing packages
A drop-in replacement for Python's standard `re` module that adds advanced regex features like nested sets, fuzzy matching, lookaround in conditionals, and full Unicode case-folding while maintaining backward compatibility.
pyparsing provides a library for building text parsers directly in Python code using composable grammar classes, handling quoted strings, whitespace variation, and embedded comments without regex or lex/yacc.
Install it if you need to parse text or define grammars programmatically.
fonttools manipulates font files in multiple formats (TrueType, OpenType, AFM, Type 1, Mac-specific) and includes TTX, a tool to convert fonts to and from XML text format.
Install it if you need to read, write, or manipulate fonts programmatically or via the TTX command-line tool.
Docutils converts plaintext documentation in reStructuredText format into multiple output formats including HTML, XML, and LaTeX using a modular processing system.
RapidFuzz provides fast fuzzy string matching using Levenshtein Distance and related metrics, implemented mostly in C++ with Python bindings for rapid similarity scoring and approximate string matching.
Install it if you need fuzzy string matching; it's a solid replacement for FuzzyWuzzy with better licensing and performance.
tinycss2 parses CSS strings into token and block objects, and generates CSS strings from those objects, following the CSS Syntax Level 3 specification without enforcing specific properties or values.
Install it if your project requires CSS tokenization or syntax manipulation.
See also json-stream · sqlfluffrs · fastokens · pytokens · moviepilot-rust · tokenizers · rustworkx · snowpipe-streaming · toons · retworkx