tdigest

What it is and what it does

tdigest is a Python implementation of Ted Dunning's t-digest algorithm, a probabilistic data structure designed to compute accurate percentiles, quantiles, and trimmed means from streaming or distributed datasets. Rather than storing all raw data points, it maintains a compact summary using centroids, allowing it to serialize to under 10kB and merge results from multiple data sources—making it particularly useful in map-reduce and distributed computing contexts.

The package provides methods to update the digest sequentially or in batches, query percentiles and cumulative distribution functions, compress the internal structure to reduce memory, and serialize/deserialize to and from Python dictionaries for storage or transmission. It depends on accumulation-tree and pyudorandom for its core operations.

Use it for:

• Computing percentiles on large streaming datasets without storing all raw values in memory.
• Aggregating statistics across distributed systems by merging multiple t-digests from different nodes.
• Estimating quantiles and trimmed means in map-reduce pipelines where data is too large to centralize.
• Serializing statistical summaries of datasets for transmission or storage with minimal overhead.
• Calculating medians and percentile ranges for real-time monitoring or analytics applications.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Implements Ted Dunning's t-digest data structure for efficient percentile and quantile estimation from streaming or distributed data, with support for merging digests and compact serialization.

Yes, if you need percentile estimation on streaming or distributed data and can tolerate an abandoned package. The algorithm is well-established and the implementation is stable; no known vulnerabilities exist. However, expect no bug fixes or updates—verify that accumulation-tree and pyudorandom remain compatible with your environment, and test accuracy for your specific use case before relying on it in production.

Install

tdigest on PyPI

pip install tdigest

uv add tdigest

poetry add tdigest

Installing tdigest

Before you install

Low install friction with two pure-Python wheels available. However, the package is abandoned—last release was 2019-05-07 and last commit 2023-05-04—so no active maintenance or security updates should be expected.

License in practice

MIT license is permissive, allowing commercial and private use with minimal restrictions; you may use, modify, and distribute this package freely provided you include the license notice.

Quickstart

from tdigest import TDigest

digest = TDigest()
digest.update(0.5)
digest.update(0.3)
print(digest.percentile(50))  # median

Verify before relying

• Whether the package's two runtime dependencies (accumulation-tree, pyudorandom) are actively maintained or have known issues.
• Whether accuracy of percentile estimates meets requirements for your specific use case and data distribution.
• Current compatibility with modern Python versions beyond what the classifiers indicate.

Package facts

Evidence: tdigest-0.5.2.2-py2.py3-none-any.whl; tdigest-0.5.2.2-py3-none-any.whl

Keywords: percentile, median, probabilistic data structure, quantile, distributed, qdigest, tdigest, streaming, pyspark

Tags

More Scientific/Engineering packages

Further reading