semhash

What it is and what it does

SemHash is a semantic deduplication and filtering library that identifies and removes duplicate or near-duplicate records from datasets by computing embeddings and performing efficient similarity search. It works with text out of the box using fast model2vec embeddings, and supports images, audio, and other modalities through custom encoders. The library handles both single-dataset operations (cleaning a training set in place) and cross-dataset operations (removing records from one dataset that are semantically similar to another, useful for eliminating train/test leakage). It can work with simple lists or complex multi-column datasets, and provides inspection tools to understand deduplication decisions—including the ability to view duplicate clusters, compute statistics like duplicate ratio, and rethreshold results without recomputing embeddings.

Under the hood, SemHash relies on model2vec for fast text embeddings and vicinity for efficient approximate nearest-neighbor search, making it practical for large datasets. The library returns structured result objects that expose both the cleaned data and metadata about what was removed, allowing users to tune thresholds and inspect edge cases. It is lightweight with minimal dependencies and supports Python 3.10 and later.

Use it for:

• Clean training datasets by removing duplicate or near-duplicate text records before model training.
• Detect and remove train/test leakage by deduplicating test data against training data with a configurable similarity threshold.
• Filter outliers from image or text datasets to improve data quality for machine learning pipelines.
• Find representative samples from a large dataset to create a smaller, diverse subset for annotation or analysis.
• Deduplicate multi-column datasets like QA pairs by combining similarity across multiple fields.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

SemHash deduplicates, filters outliers from, and finds representative samples in text, image, and multimodal datasets using semantic similarity with fast embeddings and efficient nearest-neighbor search.

Yes. SemHash is actively maintained, has no known vulnerabilities, installs with low friction, and solves a concrete problem—semantic deduplication—with a clean API and inspection tools. It is well-suited for data cleaning workflows in machine learning and is permissively licensed. The main consideration is that it requires Python 3.10+, and custom modalities beyond text require bringing your own encoder.

Install

semhash on PyPI

pip install semhash

uv add semhash

poetry add semhash

Installing semhash

Before you install

Low friction: pure Python wheel with four runtime dependencies (model2vec, vicinity, frozendict, pyversity). Active maintenance with recent commits and 958 repository stars.

License in practice

MIT License permits unrestricted use, modification, and distribution with minimal restrictions—suitable for commercial and open-source projects.

Quickstart

pip install semhash

from semhash import SemHash

texts = ["example text 1", "example text 2"]
semhash = SemHash.from_records(records=texts)
result = semhash.self_deduplicate()
deduplicated = result.selected

Requires Python 3.10 or later. Text embedding uses model2vec; custom modalities require a compatible encoder passed to the model parameter.

Verify before relying

• Performance benchmarks and throughput for large-scale datasets (millions of records) are referenced but not detailed in the excerpt.
• Memory footprint and resource requirements for different dataset sizes and modalities.
• Specific similarity thresholds and their typical effects on deduplication ratios across different data types.

Package facts

Evidence: semhash-0.4.1-py3-none-any.whl

Tags

More Libraries packages