{"categories":[{"label":"Artificial Intelligence","url":"https://skillfed.io/packages/category/scientific-engineering-artificial-intelligence/4"}],"enrichment":{"capability":"Provides a PySpark-compatible implementation of the Hierarchical Navigable Small World (HNSW) algorithm for approximate nearest neighbor search with support for multiple distance metrics.","skillfed_tags":["vector-search","distributed-computing","similarity-search"],"use_cases":["Build a recommendation engine in PySpark that finds similar items based on embedding vectors.","Perform large-scale similarity search on text embeddings or image feature vectors in a distributed Spark cluster.","Index high-dimensional data incrementally and query for approximate nearest neighbors without full-table scans.","Implement semantic search on document embeddings using multiple distance metrics tailored to your data.","Prototype or deploy vector search pipelines that integrate with existing PySpark ETL workflows."],"what_it_does":"pyspark-hnsw is a PySpark wrapper around a Java/Scala implementation of the HNSW (Hierarchical Navigable Small World) algorithm, a method for performing approximate nearest neighbor search on high-dimensional data. It allows you to build and query indexes that find similar vectors efficiently without scanning every item in a dataset. The package supports seven pre-packaged distance metrics (euclidean, cosine, manhattan, inner product, correlation, canberra, and bray curtis dissimilarity) and exposes a flexible interface for custom distance functions.\n\nThe index is thread-safe, serializable, and supports incremental insertion and experimental deletion. It integrates with PySpark's distributed computing model, making it suitable for large-scale similarity search workloads. However, the package has not seen a release since late 2022, and its Python version support is unspecified, so compatibility with modern Python environments should be verified before use.","worth_installing":"Yes, if you need approximate nearest neighbor search in PySpark and are willing to verify Python compatibility and license terms. The low install friction and lack of runtime dependencies make it straightforward to add. However, the aging maintenance status (no release since 2022-12-30) and unclear license warrant checking the repository before production use. Not recommended for new projects unless you have specific PySpark integration requirements that outweigh the maintenance risk."},"id":"pyspark-hnsw","links":{"html":"https://skillfed.io/packages/pyspark-hnsw","md":"https://skillfed.io/packages/pyspark-hnsw.md","pypi":"https://pypi.org/project/pyspark-hnsw/"},"maintenance":{"status":"aging"},"meta":{"latest_release":"2022-12-30","license_spdx":null,"license_treatment":"unclear","name":"pyspark-hnsw","python_support":"unspecified","summary":null},"popularity":{"monthly_downloads":685650,"position":5352,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"1.1.0"}