{"categories":[{"label":"Distributed Computing","url":"https://skillfed.io/packages/category/system-distributed-computing/2"}],"enrichment":{"capability":"Integrates PySpark with Dagster's data pipeline orchestration, enabling you to define and run Spark-based data assets within Dagster's declarative asset model.","skillfed_tags":["spark-integration","orchestration","data-assets"],"use_cases":["Orchestrate Spark jobs as part of a larger Dagster asset graph with lineage tracking and observability.","Define reusable Spark transformations as Dagster assets that can be tested and versioned in code.","Integrate Spark-based ETL workloads with non-Spark data tasks in a single declarative pipeline.","Deploy Spark jobs to production with Dagster's multi-tenant orchestration engine and monitoring."],"what_it_does":"Dagster-pyspark is a Dagster integration package that brings PySpark into Dagster's asset-oriented orchestration model. It provides components to define data assets that run on Spark, allowing you to declare transformations as Python functions while leveraging Spark's distributed computing. The package sits between Dagster (the orchestrator) and PySpark (the execution engine), handling resource configuration and job submission.\n\nYou use it when you have Spark workloads you want to orchestrate alongside other data assets in Dagster. It depends on dagster and pyspark as runtime dependencies, so you need both installed. The package is part of the broader Dagster ecosystem and is actively maintained alongside the main framework.","worth_installing":"Yes, if you are already using Dagster and need to run PySpark workloads. It is actively maintained, has no known vulnerabilities, installs with low friction, and carries a permissive license. Install it only if you have Spark jobs to orchestrate; it adds no value as a standalone package."},"id":"dagster-pyspark","links":{"html":"https://skillfed.io/packages/dagster-pyspark","md":"https://skillfed.io/packages/dagster-pyspark.md","pypi":"https://pypi.org/project/dagster-pyspark/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-08-14","license_spdx":"Apache-2.0","license_treatment":"permissive","name":"dagster-pyspark","python_support":"supports_current","summary":"Package for PySpark Dagster framework components."},"popularity":{"monthly_downloads":632109,"position":5655,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.29.18"}