{"categories":[{"label":"Scientific/Engineering","url":"https://skillfed.io/packages/category/scientific-engineering/6"}],"enrichment":{"capability":"Distributes scikit-learn model training and hyperparameter search across Spark clusters, converting Spark DataFrames to numpy arrays or sparse matrices for parallel processing.","skillfed_tags":["distributed-ml","archived"],"use_cases":["Distribute hyperparameter grid search across a Spark cluster for small-to-medium datasets that fit in worker memory.","Convert Spark DataFrames to numpy arrays for use with scikit-learn models in a distributed training pipeline.","Run parallel cross-validation of multiple scikit-learn estimators on a Spark cluster without rewriting model code.","Prototype distributed machine learning workflows before migrating to Spark MLlib for true distributed algorithms."],"what_it_does":"spark-sklearn bridges Apache Spark and scikit-learn to run hyperparameter search and model evaluation in parallel across a Spark cluster. It converts Spark DataFrames into numpy arrays or sparse matrices and distributes grid-search cross-validation tasks, acting as a distributed analog to scikit-learn's built-in multicore joblib backend. The package is designed for small datasets that fit in memory but benefit from parallel search; for larger datasets that don't fit in memory, the documentation recommends Spark MLlib instead.\n\nThe package provides a drop-in replacement API for scikit-learn's GridSearchCV that accepts a Spark context and distributes the search work across cluster nodes. It does not distribute individual learning algorithms themselves\u2014only the task-level parallelism of trying different hyperparameter combinations. The project is no longer maintained: the repository was archived in 2019, and the last release (0.3.0) dates to January 2019, making it incompatible with modern scikit-learn and Python versions.","worth_installing":"No. The package is abandoned (last release 2019-01-30, repository archived), incompatible with scikit-learn versions after 0.19, and untested with modern Python 3.x. High install friction (external Spark dependency, version pinning) combined with no maintenance path makes it unsuitable for new projects. Consider Spark MLlib or modern distributed ML frameworks instead."},"id":"spark-sklearn","links":{"html":"https://skillfed.io/packages/spark-sklearn","md":"https://skillfed.io/packages/spark-sklearn.md","pypi":"https://pypi.org/project/spark-sklearn/"},"maintenance":{"status":"abandoned"},"meta":{"latest_release":"2019-01-30","license_spdx":null,"license_treatment":"permissive","name":"spark-sklearn","python_support":"unspecified","summary":"Integration tools for running scikit-learn on Spark"},"popularity":{"monthly_downloads":196561,"position":9781,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.3.0"}