{"categories":[{"label":"Distributed Computing","url":"https://skillfed.io/packages/category/system-distributed-computing/3"}],"enrichment":{"capability":"RayDP runs Apache Spark on Ray and integrates Spark with AI libraries, enabling you to build distributed data and machine learning pipelines in a single Python program.","skillfed_tags":["spark-ray-integration","distributed-ml-pipeline","data-processing"],"use_cases":["Build end-to-end ML pipelines combining Spark data processing with PyTorch or TensorFlow training in a single Python script.","Run on-demand Spark jobs in cloud environments without manually provisioning a separate Spark cluster.","Convert Spark DataFrames to Ray Datasets for distributed training with XGBoost or Horovod on Ray.","Unify data processing and model serving on a single Ray cluster managed by an ML infrastructure team.","Scale data science workflows from laptop to cloud without rewriting code or managing multiple cluster types."],"what_it_does":"RayDP bridges Spark and Ray, allowing you to run Spark jobs as Ray actors on a single Ray cluster instead of managing separate Spark and Ray infrastructure. This eliminates the operational overhead of maintaining two clusters and removes the latency of exchanging data through external storage systems.\n\nThe package provides three main integration patterns: direct Spark-on-Ray execution via `raydp.init_spark()`, bidirectional conversion between Spark DataFrames and Ray Datasets for consumption by XGBoost or Ray Train, and high-level Estimator APIs (TorchEstimator, TFEstimator) that wrap Ray Train to train PyTorch or TensorFlow models directly on Spark DataFrames. It depends on numpy, pandas, psutil, pyarrow, ray, pyspark, and protobuf.","worth_installing":"Yes, if you are building distributed data and AI pipelines and want to avoid managing separate Spark and Ray clusters. The active maintenance, permissive license, and low install friction make it a practical choice. Requires Java setup and familiarity with both Spark and Ray APIs; not suitable if you need only Spark or only Ray in isolation."},"id":"raydp","links":{"html":"https://skillfed.io/packages/raydp","md":"https://skillfed.io/packages/raydp.md","pypi":"https://pypi.org/project/raydp/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-03-12","license_spdx":null,"license_treatment":"permissive","name":"raydp","python_support":"supports_current","summary":"RayDP: Distributed Data Processing on Ray"},"popularity":{"monthly_downloads":108521,"position":12553,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"1.6.5"}