datasketch
Probabilistic data structures for processing and searching very large datasets
Decision gist · record as of 2026-08-14
Yes. datasketch is production-stable (Development Status 5), actively maintained, has no known vulnerabilities, and solves a real problem—approximate similarity and cardinality at scale with low install friction. The MIT license is unrestricted. Version 2.0.0's breaking changes to MinHash serialization are a one-time migration cost but necessary for correctness and performance; the legacy scheme option eases interop. Install if you need fast approximate set operations on large data.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.9 or above; numpy and scipy must be installed (pip handles this automatically).
- Low friction: pure Python wheel with only numpy and scipy as runtime dependencies.
- Active maintenance—last release 40 days ago, repository actively developed with recent commits.
License · maintenance · safety
MIT (permissive) — MIT license is permissive; you can use, modify, and distribute this package freely with minimal restrictions.
last release 2026-07-05 (40 days) · last repo commit 2026-08-09 · 2,952 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 6,879,885 downloads/mo, #1,824 on PyPI
Alternatives
Verify before relying
pip install datasketch
from datasketch import MinHash
m1 = MinHash()
m1.update(b'apple')
m1.update(b'banana')
m2 = MinHash()
m2.update(b'banana')
m2.update(b'cherry')
similarity = m1.jaccard(m2)
print(f"Jaccard similarity: {similarity}")- Whether version 2.0.0's breaking changes to MinHash serialization format significantly impact existing deployments or are easily managed via the legacy scheme option.
- Performance characteristics of the new affine32 scheme versus legacy across different dataset sizes and similarity distributions.
- Availability and maturity of Redis and Cassandra storage backends for production scale-out scenarios.
What it is and what it does
datasketch implements a collection of probabilistic sketching algorithms—MinHash, Weighted MinHash, HyperLogLog, and HyperLogLog++—that compress large sets into compact signatures for fast approximate computation. It's built on numpy and scipy and designed to handle datasets too large for exact computation by trading small accuracy loss for dramatic speed and memory gains. The package also provides indexes (MinHash LSH, LSH Forest, LSH Ensemble, HNSW) that enable sub-linear query time for similarity search and top-k retrieval.
Version 2.0.0 changes the default MinHash permutation scheme to affine32, which reduces memory by half and speeds updates roughly 4x while fixing a similarity over-estimation bias on large sets. Existing persisted sketches must be rebuilt or interoperated via a legacy scheme flag. The package supports Python 3.9+ and optional Redis or Cassandra backends for distributed storage of LSH indexes.
Use it for
- Estimate Jaccard similarity between large text documents or sets without computing exact overlap.
- Count approximate cardinality of massive streams or datasets using HyperLogLog sketches.
- Build fast similarity search indexes over millions of items using MinHash LSH for threshold queries.
- Retrieve top-k most similar items from a large corpus using LSH Forest or HNSW indexes.
- Detect near-duplicate content or deduplication in data mining pipelines with minimal memory.
- Estimate containment relationships between sets at scale using LSH Ensemble.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
datasketch is production-stable (Development Status 5), actively maintained, has no known vulnerabilities, and solves a real problem—approximate similarity and cardinality at scale with low install friction. The MIT license is unrestricted. Version 2.0.0's breaking changes to MinHash serialization are a one-time migration cost but necessary for correctness and performance; the legacy scheme option eases interop. Install if you need fast approximate set operations on large data.
Install
datasketch on PyPI
Before you install
Low friction: pure Python wheel with only numpy and scipy as runtime dependencies. Active maintenance—last release 40 days ago, repository actively developed with recent commits.
Requires Python 3.9 or above; numpy and scipy must be installed (pip handles this automatically).
License in practice
MIT license is permissive; you can use, modify, and distribute this package freely with minimal restrictions.
Quickstart
pip install datasketch
from datasketch import MinHash
m1 = MinHash()
m1.update(b'apple')
m1.update(b'banana')
m2 = MinHash()
m2.update(b'banana')
m2.update(b'cherry')
similarity = m1.jaccard(m2)
print(f"Jaccard similarity: {similarity}")Verify before relying
- Whether version 2.0.0's breaking changes to MinHash serialization format significantly impact existing deployments or are easily managed via the legacy scheme option.
- Performance characteristics of the new affine32 scheme versus legacy across different dataset sizes and similarity distributions.
- Availability and maturity of Redis and Cassandra storage backends for production scale-out scenarios.
Package facts
| License | MIT permissive |
| Python support | Supports the current Python release >=3.9 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 2 packagesnumpyscipy |
| Maintenance | Actively maintained 40 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 6,879,885 / month, #1,824 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 5 - Production/StableIntended Audience :: DevelopersLicense :: OSI Approved :: MIT LicenseProgramming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.9Topic :: DatabaseTopic :: Scientific/Engineering :: Information Analysis |
Evidence: datasketch-2.0.0-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “minhash lsh indexing”
- datasketchProvides probabilistic data structures (MinHash, HyperLogLog, and…
- rensaRensa computes MinHash signatures for fast similarity estimation and…
- mhfpEncodes molecular structures as MinHash fingerprints (MHFP6) for fast…
Give your agent the search over MCP, or paste the wish link into any chat.
More Database packages
psycopg2-binary is a PostgreSQL database adapter for Python that implements the DB API 2.0 specification, enabling Python applications to connect to and query PostgreSQL databases with thread-safe concurrent operations.
Python client library for connecting to and executing commands against Redis key-value stores, supporting both synchronous and asynchronous operations.
Install it if your application needs to interact with Redis; the only prerequisite is a running Redis server instance.
YDB Python SDK is the official client library for connecting to and querying YDB databases from Python applications.
Install it if you need to connect Python applications to YDB databases.
Connects Python applications to Snowflake data warehouses using the DB API 2.0 specification, enabling SQL queries, data transfers, and warehouse operations.
sqlparse tokenizes SQL text into a tree of statements, clauses, and expressions, and provides functions to split scripts, format queries, and inspect parsed tokens without validating dialect or syntax.
Install it if you need to manipulate, format, or analyze SQL text programmatically.
Provides base adapter protocols and shared functionality that database adapters use to integrate with dbt-core, handling connections, dialect translation, relation caching, and core interface management.
See also datasketches · HLL · rensa · whylogs-sketching · madoka · ddsketch · pyprobables · mhfp · pyspark-hnsw · usearch