dagster-pyspark

What it is and what it does

Dagster-pyspark is a Dagster integration package that brings PySpark into Dagster's asset-oriented orchestration model. It provides components to define data assets that run on Spark, allowing you to declare transformations as Python functions while leveraging Spark's distributed computing. The package sits between Dagster (the orchestrator) and PySpark (the execution engine), handling resource configuration and job submission.

You use it when you have Spark workloads you want to orchestrate alongside other data assets in Dagster. It depends on dagster and pyspark as runtime dependencies, so you need both installed. The package is part of the broader Dagster ecosystem and is actively maintained alongside the main framework.

Use it for:

• Orchestrate Spark jobs as part of a larger Dagster asset graph with lineage tracking and observability.
• Define reusable Spark transformations as Dagster assets that can be tested and versioned in code.
• Integrate Spark-based ETL workloads with non-Spark data tasks in a single declarative pipeline.
• Deploy Spark jobs to production with Dagster's multi-tenant orchestration engine and monitoring.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Integrates PySpark with Dagster's data pipeline orchestration, enabling you to define and run Spark-based data assets within Dagster's declarative asset model.

Yes, if you are already using Dagster and need to run PySpark workloads. It is actively maintained, has no known vulnerabilities, installs with low friction, and carries a permissive license. Install it only if you have Spark jobs to orchestrate; it adds no value as a standalone package.

Install

dagster-pyspark on PyPI

pip install dagster-pyspark

uv add dagster-pyspark

poetry add dagster-pyspark

Installing dagster-pyspark

Before you install

Low friction install with a pure-Python wheel. Dagster itself is actively maintained with recent releases and strong community engagement. Requires Python 3.10 or later.

License in practice

Apache-2.0 permissive license allows commercial use, modification, and distribution with minimal restrictions—suitable for most production and proprietary contexts.

Quickstart

pip install dagster-pyspark

import dagster as dg
from dagster_pyspark import pyspark_resource

@dg.asset
def my_spark_asset() -> None:
    pass

Requires Python 3.10 or later and a working PySpark installation (pyspark is a runtime dependency).

Verify before relying

• Whether this package provides Spark-specific resource types or ops beyond what base Dagster offers
• Whether it includes built-in support for Spark SQL, DataFrame operations, or just job submission
• Performance characteristics and scalability limits when orchestrating large Spark clusters

Package facts

Evidence: dagster_pyspark-0.29.18-py3-none-any.whl

Tags

More Distributed Computing packages