dagster-databricks

What it is and what it does

dagster-databricks is a Dagster integration library that bridges Dagster's declarative data orchestration framework with Databricks infrastructure. It provides Databricks-specific ops and resource components that let you build data pipelines as Python functions, with Dagster handling scheduling, lineage tracking, and observability while your code runs against Databricks clusters and SQL warehouses.

The package sits within Dagster's broader ecosystem—you declare data assets using Dagster's programming model, and dagster-databricks handles the Databricks-specific plumbing: authentication, cluster interaction, and job submission. It's designed for teams already using Databricks who want Dagster's asset-centric orchestration, testing, and monitoring capabilities layered on top.

Use it for:

• Build and schedule SQL transformations on Databricks Delta Lake tables, with Dagster tracking lineage and data quality.
• Orchestrate multi-step data pipelines that read from Databricks, transform in Python, and write results back to Databricks.
• Run PySpark jobs on Databricks clusters via Dagster, with centralized monitoring and error handling across your data platform.
• Define machine learning training pipelines that pull features from Databricks and log models back to a Databricks workspace.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Integrates Databricks with Dagster's data orchestration framework, enabling you to define and run data pipelines that interact with Databricks clusters and SQL warehouses.

Yes, if you are already using Dagster and Databricks and want a native integration between them. The package is actively maintained, has no known vulnerabilities, and low install friction. Install only if you need Databricks-specific orchestration components; the base dagster library alone may suffice for simpler use cases.

Install

dagster-databricks on PyPI

pip install dagster-databricks

uv add dagster-databricks

poetry add dagster-databricks

Installing dagster-databricks

Before you install

Low friction install with a wheel distribution. Actively maintained as of 2026-08-14 with no known vulnerabilities. Depends on dagster and databricks-sdk among other core libraries.

License in practice

Apache-2.0 licensed, permissive terms allowing commercial and private use with minimal restrictions.

Quickstart

pip install dagster-databricks

import dagster as dg
from dagster_databricks import databricks_resource

@dg.asset
def my_table(context):
    # Use Databricks resource to query or write data
    pass

Requires Python 3.10 or later (supports up to 3.14). Requires active Databricks workspace credentials and connection configuration.

Verify before relying

• Whether this package provides Databricks-specific ops/resources beyond what the base dagster library offers
• What specific Databricks operations (SQL, Delta Lake, jobs) are supported by the included components
• Whether additional setup or authentication configuration is needed beyond standard Databricks SDK setup

Package facts

Evidence: dagster_databricks-0.29.18-py3-none-any.whl

Tags

More Distributed Computing packages