spark-sklearn
Integration tools for running scikit-learn on Spark
Decision gist · record as of 2026-08-14
No. The package is abandoned (last release 2019-01-30, repository archived), incompatible with scikit-learn versions after 0.19, and untested with modern Python 3.x. High install friction (external Spark dependency, version pinning) combined with no maintenance path makes it unsuitable for new projects. Consider Spark MLlib or modern distributed ML frameworks instead.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Apache Spark >= 2.1.1 installed and configured separately; scikit-learn 0.18 or 0.19 (later versions incompatible); pyspark interpreter or Spark-compliant Python environment; SPARK_HOME environment variable set.
- High install friction: the package is abandoned (last release 2019-01-30, repository archived).
- It requires external Spark installation and specific scikit-learn versions (0.18 or 0.19), making setup complex.
License · maintenance · safety
Apache 2.0 (permissive) — Licensed under Apache 2.0 (permissive), which allows commercial use, modification, and distribution with minimal restrictions. This poses no legal barrier to adoption.
last release 2019-01-30 (2753 days) · last repo commit 2019-12-03 · 1,071 stars · archived
0 known vulnerabilities (OSV.dev, 2026-08-14) · 196,561 downloads/mo, #9,781 on PyPI
Alternatives
Verify before relying
pip install spark-sklearn
from sklearn import svm, datasets
from spark_sklearn import GridSearchCV
iris = datasets.load_iris()
parameters = {'kernel':('linear', 'rbf'), 'C':[1, 10]}
svr = svm.SVC(gamma='auto')
clf = GridSearchCV(sc, svr, parameters)
clf.fit(iris.data, iris.target)- Compatibility with scikit-learn versions released after 0.20 and modern Python 3.x versions.
- Whether the package works with current Spark versions (tested only up to 2.1.1 era).
- Status of the distributed sparse matrix functionality marked as experimental in the description.
What it is and what it does
spark-sklearn bridges Apache Spark and scikit-learn to run hyperparameter search and model evaluation in parallel across a Spark cluster. It converts Spark DataFrames into numpy arrays or sparse matrices and distributes grid-search cross-validation tasks, acting as a distributed analog to scikit-learn's built-in multicore joblib backend. The package is designed for small datasets that fit in memory but benefit from parallel search; for larger datasets that don't fit in memory, the documentation recommends Spark MLlib instead.
The package provides a drop-in replacement API for scikit-learn's GridSearchCV that accepts a Spark context and distributes the search work across cluster nodes. It does not distribute individual learning algorithms themselves—only the task-level parallelism of trying different hyperparameter combinations. The project is no longer maintained: the repository was archived in 2019, and the last release (0.3.0) dates to January 2019, making it incompatible with modern scikit-learn and Python versions.
Use it for
- Distribute hyperparameter grid search across a Spark cluster for small-to-medium datasets that fit in worker memory.
- Convert Spark DataFrames to numpy arrays for use with scikit-learn models in a distributed training pipeline.
- Run parallel cross-validation of multiple scikit-learn estimators on a Spark cluster without rewriting model code.
- Prototype distributed machine learning workflows before migrating to Spark MLlib for true distributed algorithms.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
No.
The package is abandoned (last release 2019-01-30, repository archived), incompatible with scikit-learn versions after 0.19, and untested with modern Python 3.x. High install friction (external Spark dependency, version pinning) combined with no maintenance path makes it unsuitable for new projects. Consider Spark MLlib or modern distributed ML frameworks instead.
Install
spark-sklearn on PyPI
Before you install
High install friction: the package is abandoned (last release 2019-01-30, repository archived). It requires external Spark installation and specific scikit-learn versions (0.18 or 0.19), making setup complex. Not maintained for modern Python or dependency versions.
Requires Apache Spark >= 2.1.1 installed and configured separately; scikit-learn 0.18 or 0.19 (later versions incompatible); pyspark interpreter or Spark-compliant Python environment; SPARK_HOME environment variable set.
License in practice
Licensed under Apache 2.0 (permissive), which allows commercial use, modification, and distribution with minimal restrictions. This poses no legal barrier to adoption.
Quickstart
pip install spark-sklearn
from sklearn import svm, datasets
from spark_sklearn import GridSearchCV
iris = datasets.load_iris()
parameters = {'kernel':('linear', 'rbf'), 'C':[1, 10]}
svr = svm.SVC(gamma='auto')
clf = GridSearchCV(sc, svr, parameters)
clf.fit(iris.data, iris.target)Verify before relying
- Compatibility with scikit-learn versions released after 0.20 and modern Python 3.x versions.
- Whether the package works with current Spark versions (tested only up to 2.1.1 era).
- Status of the distributed sparse matrix functionality marked as experimental in the description.
Package facts
| License | Apache 2.0 permissive |
| Python support | Not specified |
| Install friction | High. Source build required |
| Runtime dependencies | None |
| Maintenance | Abandoned 2,753 days since the last release |
| Last repo commit | repository archived |
| First released | |
| Downloads | 196,561 / month, #9,781 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 4 - BetaIntended Audience :: DevelopersLicense :: OSI Approved :: Apache Software LicenseNatural Language :: EnglishOperating System :: OS IndependentProgramming Language :: PythonProgramming Language :: Python :: 2.6Programming Language :: Python :: 2.7Programming Language :: Python :: 3Programming Language :: Python :: 3.2Topic :: Scientific/Engineering |
Evidence: spark-sklearn-0.3.0.tar.gz
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “spark scikit-learn integration”
- spark-sklearnDistributes scikit-learn model training and hyperparameter search…
- dtreevizdtreeviz renders decision trees from scikit-learn, XGBoost, LightGBM,…
- apache-airflow-providers-apache-sparkIntegrates Apache Spark with Apache Airflow, enabling you to…
Give your agent the search over MCP, or paste the wish link into any chat.
More Scientific/Engineering packages
NumPy provides an N-dimensional array object and a comprehensive suite of mathematical, linear algebra, Fourier transform, and random number functions for scientific computing in Python.
pandas provides fast, flexible data structures (Series and DataFrame) for loading, cleaning, transforming, and analyzing labeled or relational data in Python.
scipy provides numerical algorithms for mathematics, science, and engineering—including optimization, integration, linear algebra, Fourier transforms, signal and image processing, and ODE solvers—built on numpy arrays.
scikit-learn provides a comprehensive Python library for supervised and unsupervised machine learning, including classification, regression, clustering, dimensionality reduction, and model evaluation tools built on NumPy and SciPy.
Install it if you need to train, evaluate, or deploy supervised or unsupervised learning models.
dill extends Python's pickle module to serialize and deserialize a much wider range of Python objects, including functions, lambdas, classes, and interpreter sessions, to byte streams for storage or network transmission.
Multiprocess is an enhanced fork of Python's standard multiprocessing library that uses dill for better serialization, allowing you to spawn processes with a threading-like API and share complex objects between them.
Install it if you use multiprocessing and encounter pickle serialization limits with lambdas or complex objects.
See also joblibspark · mleap · pyspark · scikit-network · pyspark-client · pyspark-pandas · dask-ml · synapseml · repartipy · scikit-multilearn