raydp

What it is and what it does

RayDP bridges Spark and Ray, allowing you to run Spark jobs as Ray actors on a single Ray cluster instead of managing separate Spark and Ray infrastructure. This eliminates the operational overhead of maintaining two clusters and removes the latency of exchanging data through external storage systems.

The package provides three main integration patterns: direct Spark-on-Ray execution via `raydp.init_spark()`, bidirectional conversion between Spark DataFrames and Ray Datasets for consumption by XGBoost or Ray Train, and high-level Estimator APIs (TorchEstimator, TFEstimator) that wrap Ray Train to train PyTorch or TensorFlow models directly on Spark DataFrames. It depends on numpy, pandas, psutil, pyarrow, ray, pyspark, and protobuf.

Use it for:

• Build end-to-end ML pipelines combining Spark data processing with PyTorch or TensorFlow training in a single Python script.
• Run on-demand Spark jobs in cloud environments without manually provisioning a separate Spark cluster.
• Convert Spark DataFrames to Ray Datasets for distributed training with XGBoost or Horovod on Ray.
• Unify data processing and model serving on a single Ray cluster managed by an ML infrastructure team.
• Scale data science workflows from laptop to cloud without rewriting code or managing multiple cluster types.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

RayDP runs Apache Spark on Ray and integrates Spark with AI libraries, enabling you to build distributed data and machine learning pipelines in a single Python program.

Yes, if you are building distributed data and AI pipelines and want to avoid managing separate Spark and Ray clusters. The active maintenance, permissive license, and low install friction make it a practical choice. Requires Java setup and familiarity with both Spark and Ray APIs; not suitable if you need only Spark or only Ray in isolation.

Install

raydp on PyPI

pip install raydp

uv add raydp

poetry add raydp

Installing raydp

Before you install

Low friction: pure Python wheel with no compiled dependencies. Active maintenance (last commit 2026-06-10) and current Python support (3.8, 3.9, 3.10). Requires Ray and PySpark as runtime dependencies, plus Java and JAVA_HOME configuration.

License in practice

Apache 2.0 permissive license allows commercial and private use with minimal restrictions, suitable for most production environments.

Quickstart

pip install raydp

import ray
import raydp

ray.init()
spark = raydp.init_spark(app_name='Example', num_executors=2, executor_cores=2, executor_memory='4GB')
df = spark.createDataFrame([('word',)], ['text'])
df.show()
raydp.stop_spark()

Requires Java to be installed with JAVA_HOME environment variable set; Ray and PySpark must be available in the Python environment.

Verify before relying

• Performance overhead of running Spark on Ray versus native Spark clusters in production scenarios.
• Compatibility matrix with specific Ray and PySpark versions beyond the stated Python 3.6+ requirement.
• Scalability limits and resource management behavior under high-concurrency workloads.

Package facts

Evidence: raydp-1.6.5-py3-none-any.whl

Keywords: raydp, spark, ray, distributed, data-processing

Tags

More Distributed Computing packages