datasketches

What it is and what it does

DataSketches is a Python binding to the Apache DataSketches library, a collection of streaming algorithms designed to answer expensive big-data queries approximately but with mathematically proven error bounds. Instead of computing exact results (which may require prohibitive compute time and resources), sketches trade precision for speed—often delivering results orders of magnitude faster. The library includes algorithms for cardinality estimation (HLL, CPC), quantile approximation (KLL, REQ, Quantiles), frequent-item detection, set operations (Theta, Tuple), sampling (VarOpt, EBPPS), and specialized tools like count-min sketches and kernel density estimation.

The package wraps C++ implementations via nanobind, providing a Python API that closely mirrors the C++ interface. It depends on numpy and is distributed as precompiled wheels for modern Python versions and common platforms. The library is portable across languages when endianness matches, making it suitable for systems that need to serialize sketches across different environments. Use it when exact answers are infeasible but approximate results with error guarantees are acceptable—typical scenarios include real-time analytics, interactive queries on massive datasets, and distributed systems where communication overhead dominates.

Use it for:

• Estimate cardinality (distinct count) in a data stream without storing all unique values.
• Compute approximate quantiles and percentiles from unbounded data streams with controlled error.
• Identify frequent items or heavy hitters in a stream with false-positive/negative guarantees.
• Perform approximate set operations (union, intersection) on large datasets with memory efficiency.
• Implement real-time analytics dashboards where exact results are unnecessary but speed is critical.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides streaming algorithms (sketches) for approximate answers to big-data queries like cardinality estimation, quantiles, and frequent items with proven error bounds and orders-of-magnitude speed gains.

Yes, if you need approximate answers to expensive big-data queries with proven error bounds and can tolerate a 531-day release gap. The library is stable, well-maintained under Apache governance, has no known vulnerabilities, and offers a mature API. Install friction is moderate due to C++ compilation, but prebuilt wheels reduce friction for standard platforms. Not suitable if you require exact results or need active feature development.

Install

datasketches on PyPI

pip install datasketches

uv add datasketches

poetry add datasketches

Installing datasketches

Before you install

Medium install friction due to compiled C++ bindings; prebuilt wheels available across macOS, Linux, and Windows. Last release 531 days ago; maintenance status is aging.

License in practice

Apache 2.0 permissive license; precompiled binaries include nanobind (BSD license). No restrictions on commercial use or modification.

Quickstart

pip install datasketches
import datasketches
sketch = datasketches.kll_floats_sketch()
for value in data:
    sketch.update(value)
print(sketch.get_quantile(0.5))

Requires numpy as a runtime dependency; C++ bindings require a compatible platform (wheels provided for common architectures).

Verify before relying

• Whether the 531-day gap since last release indicates active maintenance or dormancy under Apache governance.
• Exact Python version support (requires_python not specified in metadata).
• Performance characteristics and memory overhead compared to alternatives for specific sketch types.

Package facts

Evidence: datasketches-5.2.0-cp310-cp310-macosx_10_14_x86_64.whl; datasketches-5.2.0-cp310-cp310-macosx_11_0_arm64.whl; datasketches-5.2.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; datasketches-5.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; datasketches-5.2.0-cp310-cp310-musllinux_1_2_aarch64.whl; datasketches-5.2.0-cp310-cp310-musllinux_1_2_x86_64.whl; datasketches-5.2.0-cp310-cp310-win_amd64.whl; datasketches-5.2.0-cp311-cp311-macosx_10_14_x86_64.whl; datasketches-5.2.0-cp311-cp311-macosx_11_0_arm64.whl; datasketches-5.2.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; datasketches-5.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; datasketches-5.2.0-cp311-cp311-musllinux_1_2_aarch64.whl; datasketches-5.2.0-cp311-cp311-musllinux_1_2_x86_64.whl; datasketches-5.2.0-cp311-cp311-win_amd64.whl; datasketches-5.2.0-cp312-cp312-macosx_10_14_x86_64.whl; datasketches-5.2.0-cp312-cp312-macosx_11_0_arm64.whl; datasketches-5.2.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; datasketches-5.2.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; datasketches-5.2.0-cp312-cp312-musllinux_1_2_aarch64.whl; datasketches-5.2.0-cp312-cp312-musllinux_1_2_x86_64.whl

Tags

More Information Analysis packages