datasketch

What it is and what it does

datasketch implements a collection of probabilistic sketching algorithms—MinHash, Weighted MinHash, HyperLogLog, and HyperLogLog++—that compress large sets into compact signatures for fast approximate computation. It's built on numpy and scipy and designed to handle datasets too large for exact computation by trading small accuracy loss for dramatic speed and memory gains. The package also provides indexes (MinHash LSH, LSH Forest, LSH Ensemble, HNSW) that enable sub-linear query time for similarity search and top-k retrieval.

Version 2.0.0 changes the default MinHash permutation scheme to affine32, which reduces memory by half and speeds updates roughly 4x while fixing a similarity over-estimation bias on large sets. Existing persisted sketches must be rebuilt or interoperated via a legacy scheme flag. The package supports Python 3.9+ and optional Redis or Cassandra backends for distributed storage of LSH indexes.

Use it for:

• Estimate Jaccard similarity between large text documents or sets without computing exact overlap.
• Count approximate cardinality of massive streams or datasets using HyperLogLog sketches.
• Build fast similarity search indexes over millions of items using MinHash LSH for threshold queries.
• Retrieve top-k most similar items from a large corpus using LSH Forest or HNSW indexes.
• Detect near-duplicate content or deduplication in data mining pipelines with minimal memory.
• Estimate containment relationships between sets at scale using LSH Ensemble.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides probabilistic data structures (MinHash, HyperLogLog, and related indexes) for fast similarity estimation and cardinality counting on large datasets with minimal memory overhead.

Yes. datasketch is production-stable (Development Status 5), actively maintained, has no known vulnerabilities, and solves a real problem—approximate similarity and cardinality at scale with low install friction. The MIT license is unrestricted. Version 2.0.0's breaking changes to MinHash serialization are a one-time migration cost but necessary for correctness and performance; the legacy scheme option eases interop. Install if you need fast approximate set operations on large data.

Install

datasketch on PyPI

pip install datasketch

uv add datasketch

poetry add datasketch

Installing datasketch

Before you install

Low friction: pure Python wheel with only numpy and scipy as runtime dependencies. Active maintenance—last release 40 days ago, repository actively developed with recent commits.

License in practice

MIT license is permissive; you can use, modify, and distribute this package freely with minimal restrictions.

Quickstart

pip install datasketch

from datasketch import MinHash

m1 = MinHash()
m1.update(b'apple')
m1.update(b'banana')

m2 = MinHash()
m2.update(b'banana')
m2.update(b'cherry')

similarity = m1.jaccard(m2)
print(f"Jaccard similarity: {similarity}")

Requires Python 3.9 or above; numpy and scipy must be installed (pip handles this automatically).

Verify before relying

• Whether version 2.0.0's breaking changes to MinHash serialization format significantly impact existing deployments or are easily managed via the legacy scheme option.
• Performance characteristics of the new affine32 scheme versus legacy across different dataset sizes and similarity distributions.
• Availability and maturity of Redis and Cassandra storage backends for production scale-out scenarios.

Package facts

Evidence: datasketch-2.0.0-py3-none-any.whl

Keywords: database, datamining

Tags

More Database packages

Further reading