spark-sklearn

What it is and what it does

spark-sklearn bridges Apache Spark and scikit-learn to run hyperparameter search and model evaluation in parallel across a Spark cluster. It converts Spark DataFrames into numpy arrays or sparse matrices and distributes grid-search cross-validation tasks, acting as a distributed analog to scikit-learn's built-in multicore joblib backend. The package is designed for small datasets that fit in memory but benefit from parallel search; for larger datasets that don't fit in memory, the documentation recommends Spark MLlib instead.

The package provides a drop-in replacement API for scikit-learn's GridSearchCV that accepts a Spark context and distributes the search work across cluster nodes. It does not distribute individual learning algorithms themselves—only the task-level parallelism of trying different hyperparameter combinations. The project is no longer maintained: the repository was archived in 2019, and the last release (0.3.0) dates to January 2019, making it incompatible with modern scikit-learn and Python versions.

Use it for:

• Distribute hyperparameter grid search across a Spark cluster for small-to-medium datasets that fit in worker memory.
• Convert Spark DataFrames to numpy arrays for use with scikit-learn models in a distributed training pipeline.
• Run parallel cross-validation of multiple scikit-learn estimators on a Spark cluster without rewriting model code.
• Prototype distributed machine learning workflows before migrating to Spark MLlib for true distributed algorithms.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Distributes scikit-learn model training and hyperparameter search across Spark clusters, converting Spark DataFrames to numpy arrays or sparse matrices for parallel processing.

No. The package is abandoned (last release 2019-01-30, repository archived), incompatible with scikit-learn versions after 0.19, and untested with modern Python 3.x. High install friction (external Spark dependency, version pinning) combined with no maintenance path makes it unsuitable for new projects. Consider Spark MLlib or modern distributed ML frameworks instead.

Install

spark-sklearn on PyPI

pip install spark-sklearn

uv add spark-sklearn

poetry add spark-sklearn

Installing spark-sklearn

Before you install

High install friction: the package is abandoned (last release 2019-01-30, repository archived). It requires external Spark installation and specific scikit-learn versions (0.18 or 0.19), making setup complex. Not maintained for modern Python or dependency versions.

License in practice

Licensed under Apache 2.0 (permissive), which allows commercial use, modification, and distribution with minimal restrictions. This poses no legal barrier to adoption.

Quickstart

pip install spark-sklearn

from sklearn import svm, datasets
from spark_sklearn import GridSearchCV
iris = datasets.load_iris()
parameters = {'kernel':('linear', 'rbf'), 'C':[1, 10]}
svr = svm.SVC(gamma='auto')
clf = GridSearchCV(sc, svr, parameters)
clf.fit(iris.data, iris.target)

Requires Apache Spark >= 2.1.1 installed and configured separately; scikit-learn 0.18 or 0.19 (later versions incompatible); pyspark interpreter or Spark-compliant Python environment; SPARK_HOME environment variable set.

Verify before relying

• Compatibility with scikit-learn versions released after 0.20 and modern Python 3.x versions.
• Whether the package works with current Spark versions (tested only up to 2.1.1 era).
• Status of the distributed sparse matrix functionality marked as experimental in the description.

Package facts

Evidence: spark-sklearn-0.3.0.tar.gz

Keywords: spark, scikit-learn, distributed computing, machine learning

Tags

More Scientific/Engineering packages