semhash
Fast Multimodal Semantic Deduplication & Filtering
Decision gist · record as of 2026-08-14
Yes. SemHash is actively maintained, has no known vulnerabilities, installs with low friction, and solves a concrete problem—semantic deduplication—with a clean API and inspection tools. It is well-suited for data cleaning workflows in machine learning and is permissively licensed. The main consideration is that it requires Python 3.10+, and custom modalities beyond text require bringing your own encoder.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.10 or later.
- Text embedding uses model2vec; custom modalities require a compatible encoder passed to the model parameter.
- Low friction: pure Python wheel with four runtime dependencies (model2vec, vicinity, frozendict, pyversity).
License · maintenance · safety
permissive license (permissive) — MIT License permits unrestricted use, modification, and distribution with minimal restrictions—suitable for commercial and open-source projects.
last release 2026-01-20 (206 days) · last repo commit 2026-05-24 · 958 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 102,699 downloads/mo, #12,853 on PyPI
Alternatives
Verify before relying
pip install semhash
from semhash import SemHash
texts = ["example text 1", "example text 2"]
semhash = SemHash.from_records(records=texts)
result = semhash.self_deduplicate()
deduplicated = result.selected- Performance benchmarks and throughput for large-scale datasets (millions of records) are referenced but not detailed in the excerpt.
- Memory footprint and resource requirements for different dataset sizes and modalities.
- Specific similarity thresholds and their typical effects on deduplication ratios across different data types.
What it is and what it does
SemHash is a semantic deduplication and filtering library that identifies and removes duplicate or near-duplicate records from datasets by computing embeddings and performing efficient similarity search. It works with text out of the box using fast model2vec embeddings, and supports images, audio, and other modalities through custom encoders. The library handles both single-dataset operations (cleaning a training set in place) and cross-dataset operations (removing records from one dataset that are semantically similar to another, useful for eliminating train/test leakage). It can work with simple lists or complex multi-column datasets, and provides inspection tools to understand deduplication decisions—including the ability to view duplicate clusters, compute statistics like duplicate ratio, and rethreshold results without recomputing embeddings.
Under the hood, SemHash relies on model2vec for fast text embeddings and vicinity for efficient approximate nearest-neighbor search, making it practical for large datasets. The library returns structured result objects that expose both the cleaned data and metadata about what was removed, allowing users to tune thresholds and inspect edge cases. It is lightweight with minimal dependencies and supports Python 3.10 and later.
Use it for
- Clean training datasets by removing duplicate or near-duplicate text records before model training.
- Detect and remove train/test leakage by deduplicating test data against training data with a configurable similarity threshold.
- Filter outliers from image or text datasets to improve data quality for machine learning pipelines.
- Find representative samples from a large dataset to create a smaller, diverse subset for annotation or analysis.
- Deduplicate multi-column datasets like QA pairs by combining similarity across multiple fields.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
SemHash is actively maintained, has no known vulnerabilities, installs with low friction, and solves a concrete problem—semantic deduplication—with a clean API and inspection tools. It is well-suited for data cleaning workflows in machine learning and is permissively licensed. The main consideration is that it requires Python 3.10+, and custom modalities beyond text require bringing your own encoder.
Install
semhash on PyPI
Before you install
Low friction: pure Python wheel with four runtime dependencies (model2vec, vicinity, frozendict, pyversity). Active maintenance with recent commits and 958 repository stars.
Requires Python 3.10 or later. Text embedding uses model2vec; custom modalities require a compatible encoder passed to the model parameter.
License in practice
MIT License permits unrestricted use, modification, and distribution with minimal restrictions—suitable for commercial and open-source projects.
Quickstart
pip install semhash
from semhash import SemHash
texts = ["example text 1", "example text 2"]
semhash = SemHash.from_records(records=texts)
result = semhash.self_deduplicate()
deduplicated = result.selectedVerify before relying
- Performance benchmarks and throughput for large-scale datasets (millions of records) are referenced but not detailed in the excerpt.
- Memory footprint and resource requirements for different dataset sizes and modalities.
- Specific similarity thresholds and their typical effects on deduplication ratios across different data types.
Package facts
| License | permissive license permissive |
| Python support | Supports the current Python release >=3.10 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 4 packagesmodel2vecvicinityfrozendictpyversity |
| Maintenance | Actively maintained 206 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 102,699 / month, #12,853 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 4 - BetaIntended Audience :: DevelopersIntended Audience :: Science/ResearchLicense :: OSI Approved :: MIT LicenseNatural Language :: EnglishProgramming Language :: Python :: 3 :: OnlyProgramming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Topic :: Scientific/Engineering :: Artificial IntelligenceTopic :: Software Development :: Libraries |
Evidence: semhash-0.4.1-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “semantic deduplication”
- semhashSemHash deduplicates, filters outliers from, and finds representative…
- gptcacheGPTCache provides semantic caching for LLM API calls, storing and…
- chromadbChroma is a vector database and search infrastructure that stores,…
Give your agent the search over MCP, or paste the wish link into any chat.
More Libraries packages
urllib3 is an HTTP client library that provides thread-safe connection pooling, SSL/TLS verification, multipart file uploads, request retries, compression support, and proxy handling for Python applications.
Requests is a Python HTTP library that simplifies sending HTTP/1.1 requests with automatic handling of headers, authentication, cookies, and response parsing.
Pluggy provides a plugin system that lets you define hook specifications and register implementations to be called in sequence, enabling extensible Python applications without tight coupling.
Install it if you're building an extensible application or framework.
Provides parsing, arithmetic, and recurrence rule computation for dates and times, with timezone support and iCalendar RFC compliance.
Install it if you need to parse flexible date strings, compute relative dates, handle timezones, or work with recurrence rules—it's the de facto choice for these tasks.
Six provides utility functions to write Python code that runs on both Python 2.7 and Python 3.3+, smoothing over language differences between the two versions.
pytest is a testing framework that lets you write test functions using plain assert statements and automatically discovers and runs them, with detailed failure reporting.
See also imagededup · rensa · splink · recordlinkage · simhash · clip-benchmark · model2vec · fiftyone-brain · cleanlab · lpips