---
id: dagster-pyspark
version: "0.29.18"
license: Apache-2.0
license_treatment: permissive
maintenance: active
---

# dagster-pyspark — Package for PySpark Dagster framework components.

License: permissive · Maintenance: active · Downloads: 632.1K/mo

## What it is and what it does

Dagster-pyspark is a Dagster integration package that brings PySpark into Dagster's asset-oriented orchestration model. It provides components to define data assets that run on Spark, allowing you to declare transformations as Python functions while leveraging Spark's distributed computing. The package sits between Dagster (the orchestrator) and PySpark (the execution engine), handling resource configuration and job submission.

You use it when you have Spark workloads you want to orchestrate alongside other data assets in Dagster. It depends on dagster and pyspark as runtime dependencies, so you need both installed. The package is part of the broader Dagster ecosystem and is actively maintained alongside the main framework.

Use it for:

- Orchestrate Spark jobs as part of a larger Dagster asset graph with lineage tracking and observability.
- Define reusable Spark transformations as Dagster assets that can be tested and versioned in code.
- Integrate Spark-based ETL workloads with non-Spark data tasks in a single declarative pipeline.
- Deploy Spark jobs to production with Dagster's multi-tenant orchestration engine and monitoring.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Integrates PySpark with Dagster's data pipeline orchestration, enabling you to define and run Spark-based data assets within Dagster's declarative asset model.

Yes, if you are already using Dagster and need to run PySpark workloads. It is actively maintained, has no known vulnerabilities, installs with low friction, and carries a permissive license. Install it only if you have Spark jobs to orchestrate; it adds no value as a standalone package.

## Install

pip install dagster-pyspark
uv add dagster-pyspark
poetry add dagster-pyspark

## Installing dagster-pyspark

Before you install: Low friction install with a pure-Python wheel. Dagster itself is actively maintained with recent releases and strong community engagement. Requires Python 3.10 or later.

License in practice: Apache-2.0 permissive license allows commercial use, modification, and distribution with minimal restrictions—suitable for most production and proprietary contexts.

Quickstart:

pip install dagster-pyspark

import dagster as dg
from dagster_pyspark import pyspark_resource

@dg.asset
def my_spark_asset() -> None:
    pass

Requires Python 3.10 or later and a working PySpark installation (pyspark is a runtime dependency).

Verify before relying:

- Whether this package provides Spark-specific resource types or ops beyond what base Dagster offers
- Whether it includes built-in support for Spark SQL, DataFrame operations, or just job submission
- Performance characteristics and scalability limits when orchestrating large Spark clusters

## Package facts

- License: Apache-2.0 (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 632.1K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

pyspark dagster integration, spark data pipeline orchestration, dagster pyspark assets, distributed spark jobs dagster, pyspark data asset framework, spark-integration, orchestration, data-assets

[View on SkillFed](https://skillfed.io/packages/dagster-pyspark) · [View on PyPI](https://pypi.org/project/dagster-pyspark/)