joblibspark

What it is and what it does

Joblibspark bridges joblib's parallel execution framework and Apache Spark, allowing you to run joblib-parallelized code on a Spark cluster instead of a single machine. It works by registering Spark as a backend that joblib can dispatch tasks to, so parallel training routines and other joblib-compatible code automatically scale across cluster nodes.

The package is lightweight—it depends only on joblib—and integrates cleanly with joblib's `parallel_backend` context manager. However, it has documented limitations: it accelerates training loops but does not parallelize model inference or feature engineering steps, which continue to run locally. This makes it most useful for training workflows where the training loop itself is the bottleneck.

Use it for:

• Distribute parallel training workloads across a Spark cluster to speed up model tuning.
• Scale joblib-based custom parallel loops to a Spark cluster without rewriting code.
• Offload CPU-intensive training tasks from a single machine to a multi-node cluster.
• Run joblib-parallelized code on Spark without changing application code.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Registers Apache Spark as a distributed backend for joblib's parallel task execution, allowing joblib-compatible code to offload work to a Spark cluster.

Yes, if you have a Spark cluster and need to scale joblib-parallelized training workloads. Install friction is low and the integration is straightforward. No, if you only do model inference or feature engineering in parallel—the package explicitly does not accelerate those. Verify that your PySpark version is compatible and that your use case fits the training-loop parallelization model.

Install

joblibspark on PyPI

pip install joblibspark

uv add joblibspark

poetry add joblibspark

Installing joblibspark

Before you install

Low friction: single runtime dependency on joblib, pure Python wheel. Repository is active with recent commits. PySpark is not bundled—you install it separately, which is typical for Spark integrations.

License in practice

Permissive Apache license; no restrictions on commercial or proprietary use.

Quickstart

pip install joblibspark

from joblibspark import register_spark
from joblib import parallel_backend

register_spark()
with parallel_backend('spark', n_jobs=3):
    # joblib-parallelized code runs on Spark cluster
    pass

Requires PySpark (not installed by default) and a running Spark cluster or local Spark environment; joblib>=0.14.

Verify before relying

• Whether the package supports Python versions beyond 3.6 and 3.7 despite classifiers listing only those.
• Current compatibility with recent PySpark versions beyond the minimum stated.
• Whether model inference parallelization limitations documented in the description have been addressed in recent releases.

Package facts

Evidence: joblibspark-0.6.0-py3-none-any.whl

Tags

More Libraries packages