{"categories":[{"label":"Database","url":"https://skillfed.io/packages/category/database"},{"label":"Information Analysis","url":"https://skillfed.io/packages/category/scientific-engineering-information-analysis"}],"enrichment":{"capability":"Provides probabilistic data structures (MinHash, HyperLogLog, and related indexes) for fast similarity estimation and cardinality counting on large datasets with minimal memory overhead.","skillfed_tags":["sketching-algorithms","similarity-search","cardinality-estimation"],"use_cases":["Estimate Jaccard similarity between large text documents or sets without computing exact overlap.","Count approximate cardinality of massive streams or datasets using HyperLogLog sketches.","Build fast similarity search indexes over millions of items using MinHash LSH for threshold queries.","Retrieve top-k most similar items from a large corpus using LSH Forest or HNSW indexes.","Detect near-duplicate content or deduplication in data mining pipelines with minimal memory.","Estimate containment relationships between sets at scale using LSH Ensemble."],"what_it_does":"datasketch implements a collection of probabilistic sketching algorithms\u2014MinHash, Weighted MinHash, HyperLogLog, and HyperLogLog++\u2014that compress large sets into compact signatures for fast approximate computation. It's built on numpy and scipy and designed to handle datasets too large for exact computation by trading small accuracy loss for dramatic speed and memory gains. The package also provides indexes (MinHash LSH, LSH Forest, LSH Ensemble, HNSW) that enable sub-linear query time for similarity search and top-k retrieval.\n\nVersion 2.0.0 changes the default MinHash permutation scheme to affine32, which reduces memory by half and speeds updates roughly 4x while fixing a similarity over-estimation bias on large sets. Existing persisted sketches must be rebuilt or interoperated via a legacy scheme flag. The package supports Python 3.9+ and optional Redis or Cassandra backends for distributed storage of LSH indexes.","worth_installing":"Yes. datasketch is production-stable (Development Status 5), actively maintained, has no known vulnerabilities, and solves a real problem\u2014approximate similarity and cardinality at scale with low install friction. The MIT license is unrestricted. Version 2.0.0's breaking changes to MinHash serialization are a one-time migration cost but necessary for correctness and performance; the legacy scheme option eases interop. Install if you need fast approximate set operations on large data."},"id":"datasketch","links":{"html":"https://skillfed.io/packages/datasketch","md":"https://skillfed.io/packages/datasketch.md","pypi":"https://pypi.org/project/datasketch/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-07-05","license_spdx":null,"license_treatment":"permissive","name":"datasketch","python_support":"supports_current","summary":"Probabilistic data structures for processing and searching very large datasets"},"popularity":{"monthly_downloads":6879885,"position":1824,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"2.0.0"}