{"categories":[{"label":"Libraries","url":"https://skillfed.io/packages/category/software-development-libraries/9"},{"label":"Artificial Intelligence","url":"https://skillfed.io/packages/category/scientific-engineering-artificial-intelligence/10"}],"enrichment":{"capability":"SemHash deduplicates, filters outliers from, and finds representative samples in text, image, and multimodal datasets using semantic similarity with fast embeddings and efficient nearest-neighbor search.","skillfed_tags":["data-cleaning","embeddings","deduplication"],"use_cases":["Clean training datasets by removing duplicate or near-duplicate text records before model training.","Detect and remove train/test leakage by deduplicating test data against training data with a configurable similarity threshold.","Filter outliers from image or text datasets to improve data quality for machine learning pipelines.","Find representative samples from a large dataset to create a smaller, diverse subset for annotation or analysis.","Deduplicate multi-column datasets like QA pairs by combining similarity across multiple fields."],"what_it_does":"SemHash is a semantic deduplication and filtering library that identifies and removes duplicate or near-duplicate records from datasets by computing embeddings and performing efficient similarity search. It works with text out of the box using fast model2vec embeddings, and supports images, audio, and other modalities through custom encoders. The library handles both single-dataset operations (cleaning a training set in place) and cross-dataset operations (removing records from one dataset that are semantically similar to another, useful for eliminating train/test leakage). It can work with simple lists or complex multi-column datasets, and provides inspection tools to understand deduplication decisions\u2014including the ability to view duplicate clusters, compute statistics like duplicate ratio, and rethreshold results without recomputing embeddings.\n\nUnder the hood, SemHash relies on model2vec for fast text embeddings and vicinity for efficient approximate nearest-neighbor search, making it practical for large datasets. The library returns structured result objects that expose both the cleaned data and metadata about what was removed, allowing users to tune thresholds and inspect edge cases. It is lightweight with minimal dependencies and supports Python 3.10 and later.","worth_installing":"Yes. SemHash is actively maintained, has no known vulnerabilities, installs with low friction, and solves a concrete problem\u2014semantic deduplication\u2014with a clean API and inspection tools. It is well-suited for data cleaning workflows in machine learning and is permissively licensed. The main consideration is that it requires Python 3.10+, and custom modalities beyond text require bringing your own encoder."},"id":"semhash","links":{"html":"https://skillfed.io/packages/semhash","md":"https://skillfed.io/packages/semhash.md","pypi":"https://pypi.org/project/semhash/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-01-20","license_spdx":null,"license_treatment":"permissive","name":"semhash","python_support":"supports_current","summary":"Fast Multimodal Semantic Deduplication & Filtering"},"popularity":{"monthly_downloads":102699,"position":12853,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.4.1"}