---
id: datasketch
version: "2.0.0"
license: MIT
license_treatment: permissive
maintenance: active
---

# datasketch — Probabilistic data structures for processing and searching very large datasets

License: permissive · Maintenance: active · Downloads: 6.9M/mo

## What it is and what it does

datasketch implements a collection of probabilistic sketching algorithms—MinHash, Weighted MinHash, HyperLogLog, and HyperLogLog++—that compress large sets into compact signatures for fast approximate computation. It's built on numpy and scipy and designed to handle datasets too large for exact computation by trading small accuracy loss for dramatic speed and memory gains. The package also provides indexes (MinHash LSH, LSH Forest, LSH Ensemble, HNSW) that enable sub-linear query time for similarity search and top-k retrieval.

Version 2.0.0 changes the default MinHash permutation scheme to affine32, which reduces memory by half and speeds updates roughly 4x while fixing a similarity over-estimation bias on large sets. Existing persisted sketches must be rebuilt or interoperated via a legacy scheme flag. The package supports Python 3.9+ and optional Redis or Cassandra backends for distributed storage of LSH indexes.

Use it for:

- Estimate Jaccard similarity between large text documents or sets without computing exact overlap.
- Count approximate cardinality of massive streams or datasets using HyperLogLog sketches.
- Build fast similarity search indexes over millions of items using MinHash LSH for threshold queries.
- Retrieve top-k most similar items from a large corpus using LSH Forest or HNSW indexes.
- Detect near-duplicate content or deduplication in data mining pipelines with minimal memory.
- Estimate containment relationships between sets at scale using LSH Ensemble.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Provides probabilistic data structures (MinHash, HyperLogLog, and related indexes) for fast similarity estimation and cardinality counting on large datasets with minimal memory overhead.

Yes. datasketch is production-stable (Development Status 5), actively maintained, has no known vulnerabilities, and solves a real problem—approximate similarity and cardinality at scale with low install friction. The MIT license is unrestricted. Version 2.0.0's breaking changes to MinHash serialization are a one-time migration cost but necessary for correctness and performance; the legacy scheme option eases interop. Install if you need fast approximate set operations on large data.

## Install

pip install datasketch
uv add datasketch
poetry add datasketch

## Installing datasketch

Before you install: Low friction: pure Python wheel with only numpy and scipy as runtime dependencies. Active maintenance—last release 40 days ago, repository actively developed with recent commits.

License in practice: MIT license is permissive; you can use, modify, and distribute this package freely with minimal restrictions.

Quickstart:

pip install datasketch

from datasketch import MinHash

m1 = MinHash()
m1.update(b'apple')
m1.update(b'banana')

m2 = MinHash()
m2.update(b'banana')
m2.update(b'cherry')

similarity = m1.jaccard(m2)
print(f"Jaccard similarity: {similarity}")

Requires Python 3.9 or above; numpy and scipy must be installed (pip handles this automatically).

Verify before relying:

- Whether version 2.0.0's breaking changes to MinHash serialization format significantly impact existing deployments or are easily managed via the legacy scheme option.
- Performance characteristics of the new affine32 scheme versus legacy across different dataset sizes and similarity distributions.
- Availability and maturity of Redis and Cassandra storage backends for production scale-out scenarios.

## Package facts

- License: MIT (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 6.9M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

jaccard similarity estimation, cardinality estimation large data, minhash lsh indexing, probabilistic data structures, hyperloglog sketch, similarity search big data, approximate set operations, sketching-algorithms, similarity-search, cardinality-estimation

[View on SkillFed](https://skillfed.io/packages/datasketch) · [View on PyPI](https://pypi.org/project/datasketch/)