---
id: dagster-spark
version: "0.29.18"
license: Apache-2.0
license_treatment: permissive
maintenance: active
---

# dagster-spark — Package for Spark Dagster framework components.

License: permissive · Maintenance: active · Downloads: 563.6K/mo

## What it is and what it does

Dagster-spark is an integration library that brings Apache Spark into Dagster's data orchestration ecosystem. It allows you to define Spark-based computations as Dagster assets—Python functions that produce data assets—and orchestrate them alongside other data operations within Dagster's declarative programming model. The package bridges Spark's distributed compute engine with Dagster's asset tracking, lineage, and observability features.

You use it to build data pipelines where some assets are computed via Spark jobs, while maintaining unified visibility and control through Dagster's web UI and orchestration engine. It integrates with Dagster's testing framework and supports the full development lifecycle from local testing to production deployment, letting you treat Spark workloads as first-class citizens in a broader data asset graph.

Use it for:

- Build and orchestrate large-scale ETL pipelines using Spark within Dagster's asset framework
- Define Spark transformations as reusable data assets with automatic lineage tracking
- Test Spark-based computations locally during development before deploying to production
- Monitor and observe Spark job execution as part of a unified Dagster data platform
- Combine Spark processing with other data tools in a single orchestrated workflow

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Dagster-spark integrates Apache Spark with Dagster's data orchestration framework, enabling you to define and run Spark-based data assets and pipelines within Dagster's declarative programming model.

Yes, if you are already using Dagster and need to integrate Spark workloads. The package is actively maintained, has low install friction, carries no known vulnerabilities, and is permissively licensed. It is worth installing as a bridge between Spark and Dagster's orchestration model, though you should verify that your target Spark version is supported.

## Install

pip install dagster-spark
uv add dagster-spark
poetry add dagster-spark

## Installing dagster-spark

Before you install: Low friction installation as a pure Python wheel with a single runtime dependency on dagster. Actively maintained with a recent release and no known vulnerabilities.

License in practice: Apache 2.0 licensed under a permissive model, allowing commercial use, modification, and distribution with minimal restrictions.

Quickstart:

pip install dagster-spark

import dagster as dg
from dagster_spark import spark_resource

@dg.asset
def my_spark_asset(context) -> None:
    spark = context.resources.spark
    # Use spark session for Spark operations

Requires Python 3.10 or later (supports up to 3.14); Spark runtime must be available in your environment.

Verify before relying:

- Specific Spark versions supported or tested by this integration
- Whether the package handles Spark cluster deployment or only local/existing clusters
- Performance characteristics or scaling limits for large Spark workloads

## Package facts

- License: Apache-2.0 (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 563.6K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

spark integration dagster, orchestrate spark jobs, spark data pipeline framework, dagster spark assets, distributed data processing orchestration, spark cluster management, data asset orchestration, spark-integration, data-orchestration, etl

[View on SkillFed](https://skillfed.io/packages/dagster-spark) · [View on PyPI](https://pypi.org/project/dagster-spark/)