--- id: pyspark-hnsw version: "1.1.0" license: unclear license_treatment: unclear maintenance: aging --- # pyspark-hnsw License: unclear · Maintenance: aging · Downloads: 685.6K/mo ## What it is and what it does pyspark-hnsw is a PySpark wrapper around a Java/Scala implementation of the HNSW (Hierarchical Navigable Small World) algorithm, a method for performing approximate nearest neighbor search on high-dimensional data. It allows you to build and query indexes that find similar vectors efficiently without scanning every item in a dataset. The package supports seven pre-packaged distance metrics (euclidean, cosine, manhattan, inner product, correlation, canberra, and bray curtis dissimilarity) and exposes a flexible interface for custom distance functions. The index is thread-safe, serializable, and supports incremental insertion and experimental deletion. It integrates with PySpark's distributed computing model, making it suitable for large-scale similarity search workloads. However, the package has not seen a release since late 2022, and its Python version support is unspecified, so compatibility with modern Python environments should be verified before use. Use it for: - Build a recommendation engine in PySpark that finds similar items based on embedding vectors. - Perform large-scale similarity search on text embeddings or image feature vectors in a distributed Spark cluster. - Index high-dimensional data incrementally and query for approximate nearest neighbors without full-table scans. - Implement semantic search on document embeddings using multiple distance metrics tailored to your data. - Prototype or deploy vector search pipelines that integrate with existing PySpark ETL workflows. ## Worth the install? AI-flagged interpretation of the facts above — verify before relying. Provides a PySpark-compatible implementation of the Hierarchical Navigable Small World (HNSW) algorithm for approximate nearest neighbor search with support for multiple distance metrics. Yes, if you need approximate nearest neighbor search in PySpark and are willing to verify Python compatibility and license terms. The low install friction and lack of runtime dependencies make it straightforward to add. However, the aging maintenance status (no release since 2022-12-30) and unclear license warrant checking the repository before production use. Not recommended for new projects unless you have specific PySpark integration requirements that outweigh the maintenance risk. ## Install pip install pyspark-hnsw uv add pyspark-hnsw poetry add pyspark-hnsw ## Installing pyspark-hnsw Before you install: Low install friction with pure Python wheels available. Maintenance status is aging—last release was 2022-12-30, though the repository remains active with a recent commit on 2025-03-23 and modest community interest (303 stars). License in practice: License treatment is unclear; no SPDX identifier or raw license text is available in the package metadata. Verify the actual license terms in the repository before adopting this package in a commercial or restricted context. Quickstart: pip install pyspark-hnsw from pyspark_hnsw import HnswIndex index = HnswIndex() index.add(item_id, vector, distance_metric='euclidean') Requires PySpark and a compatible Java/Scala runtime environment; Python version support is unspecified. Verify before relying: - Actual license terms and compliance implications—metadata shows no SPDX or raw license field. - Minimum Python version and specific PySpark version compatibility requirements. - Whether the package is actively maintained or in maintenance-only mode given the aging status and release gap. ## Package facts - License: not declared (unclear) - Python support: unspecified - Install friction: low - Maintenance: aging - Downloads: 685.6K/month (top 15,000 on PyPI) - Known vulnerabilities: none known ## Tags approximate nearest neighbor search, hnsw algorithm pyspark, vector similarity search, hierarchical navigable small world, distance metric nearest neighbors, spark approximate search, embedding similarity, vector-search, distributed-computing, similarity-search [View on SkillFed](https://skillfed.io/packages/pyspark-hnsw) · [View on PyPI](https://pypi.org/project/pyspark-hnsw/)