pyspark-hnsw

What it is and what it does

pyspark-hnsw is a PySpark wrapper around a Java/Scala implementation of the HNSW (Hierarchical Navigable Small World) algorithm, a method for performing approximate nearest neighbor search on high-dimensional data. It allows you to build and query indexes that find similar vectors efficiently without scanning every item in a dataset. The package supports seven pre-packaged distance metrics (euclidean, cosine, manhattan, inner product, correlation, canberra, and bray curtis dissimilarity) and exposes a flexible interface for custom distance functions.

The index is thread-safe, serializable, and supports incremental insertion and experimental deletion. It integrates with PySpark's distributed computing model, making it suitable for large-scale similarity search workloads. However, the package has not seen a release since late 2022, and its Python version support is unspecified, so compatibility with modern Python environments should be verified before use.

Use it for:

• Build a recommendation engine in PySpark that finds similar items based on embedding vectors.
• Perform large-scale similarity search on text embeddings or image feature vectors in a distributed Spark cluster.
• Index high-dimensional data incrementally and query for approximate nearest neighbors without full-table scans.
• Implement semantic search on document embeddings using multiple distance metrics tailored to your data.
• Prototype or deploy vector search pipelines that integrate with existing PySpark ETL workflows.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides a PySpark-compatible implementation of the Hierarchical Navigable Small World (HNSW) algorithm for approximate nearest neighbor search with support for multiple distance metrics.

Yes, if you need approximate nearest neighbor search in PySpark and are willing to verify Python compatibility and license terms. The low install friction and lack of runtime dependencies make it straightforward to add. However, the aging maintenance status (no release since 2022-12-30) and unclear license warrant checking the repository before production use. Not recommended for new projects unless you have specific PySpark integration requirements that outweigh the maintenance risk.

Install

pyspark-hnsw on PyPI

pip install pyspark-hnsw

uv add pyspark-hnsw

poetry add pyspark-hnsw

Installing pyspark-hnsw

Before you install

Low install friction with pure Python wheels available. Maintenance status is aging—last release was 2022-12-30, though the repository remains active with a recent commit on 2025-03-23 and modest community interest (303 stars).

License in practice

License treatment is unclear; no SPDX identifier or raw license text is available in the package metadata. Verify the actual license terms in the repository before adopting this package in a commercial or restricted context.

Quickstart

pip install pyspark-hnsw

from pyspark_hnsw import HnswIndex

index = HnswIndex()
index.add(item_id, vector, distance_metric='euclidean')

Requires PySpark and a compatible Java/Scala runtime environment; Python version support is unspecified.

Verify before relying

• Actual license terms and compliance implications—metadata shows no SPDX or raw license field.
• Minimum Python version and specific PySpark version compatibility requirements.
• Whether the package is actively maintained or in maintenance-only mode given the aging status and release gap.

Package facts

Evidence: pyspark_hnsw-1.1.0-py2-none-any.whl; pyspark_hnsw-1.1.0-py3-none-any.whl

Tags

More Artificial Intelligence packages