dagster-spark

What it is and what it does

Dagster-spark is an integration library that brings Apache Spark into Dagster's data orchestration ecosystem. It allows you to define Spark-based computations as Dagster assets—Python functions that produce data assets—and orchestrate them alongside other data operations within Dagster's declarative programming model. The package bridges Spark's distributed compute engine with Dagster's asset tracking, lineage, and observability features.

You use it to build data pipelines where some assets are computed via Spark jobs, while maintaining unified visibility and control through Dagster's web UI and orchestration engine. It integrates with Dagster's testing framework and supports the full development lifecycle from local testing to production deployment, letting you treat Spark workloads as first-class citizens in a broader data asset graph.

Use it for:

• Build and orchestrate large-scale ETL pipelines using Spark within Dagster's asset framework
• Define Spark transformations as reusable data assets with automatic lineage tracking
• Test Spark-based computations locally during development before deploying to production
• Monitor and observe Spark job execution as part of a unified Dagster data platform
• Combine Spark processing with other data tools in a single orchestrated workflow

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Dagster-spark integrates Apache Spark with Dagster's data orchestration framework, enabling you to define and run Spark-based data assets and pipelines within Dagster's declarative programming model.

Yes, if you are already using Dagster and need to integrate Spark workloads. The package is actively maintained, has low install friction, carries no known vulnerabilities, and is permissively licensed. It is worth installing as a bridge between Spark and Dagster's orchestration model, though you should verify that your target Spark version is supported.

Install

dagster-spark on PyPI

pip install dagster-spark

uv add dagster-spark

poetry add dagster-spark

Installing dagster-spark

Before you install

Low friction installation as a pure Python wheel with a single runtime dependency on dagster. Actively maintained with a recent release and no known vulnerabilities.

License in practice

Apache 2.0 licensed under a permissive model, allowing commercial use, modification, and distribution with minimal restrictions.

Quickstart

pip install dagster-spark

import dagster as dg
from dagster_spark import spark_resource

@dg.asset
def my_spark_asset(context) -> None:
    spark = context.resources.spark
    # Use spark session for Spark operations

Requires Python 3.10 or later (supports up to 3.14); Spark runtime must be available in your environment.

Verify before relying

• Specific Spark versions supported or tested by this integration
• Whether the package handles Spark cluster deployment or only local/existing clusters
• Performance characteristics or scaling limits for large Spark workloads

Package facts

Evidence: dagster_spark-0.29.18-py3-none-any.whl

Tags

More Distributed Computing packages