---
id: dbt-spark
version: "1.11.0"
license: unclear
license_treatment: permissive
maintenance: active
---

# dbt-spark — The Apache Spark adapter plugin for dbt

License: permissive · Maintenance: active · Downloads: 6.5M/mo

## What it is and what it does

dbt-spark is a plugin that connects dbt (a data transformation tool built on SQL and YAML) to Apache Spark, allowing data engineers and analysts to apply software engineering practices—version control, testing, documentation, modularity—to Spark-based data pipelines. It handles the translation of dbt's declarative transformation models into Spark SQL, managing the execution and metadata tracking within a Spark cluster or local Thrift server.

The adapter sits between dbt-core and Spark, inheriting dbt's ecosystem (dbt-adapters, dbt-common) while adding Spark-specific connection logic and SQL dialect support. It is production-ready (Development Status 5), actively maintained, and supports modern Python versions (3.10–3.13) across Linux, macOS, and Windows.

Use it for:

- Transform raw data in a Spark warehouse using dbt models, organizing and cleansing data for downstream analytics.
- Build modular, testable data pipelines in Spark with version control and documentation via dbt's YAML and SQL interface.
- Integrate Spark-based ELT workflows into a dbt Cloud or dbt Core project alongside other warehouse adapters.
- Run local Spark development and testing using the docker-compose Thrift server setup for rapid iteration.
- Manage Hive Metastore schemas and table lineage through dbt's metadata layer on top of Spark.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

dbt-spark is the Apache Spark adapter for dbt, enabling data transformation workflows in Spark using dbt's SQL and YAML-based modeling practices.

Yes. dbt-spark is production-ready, actively maintained, has no known vulnerabilities, low install friction, and a permissive license. Install it if you use Apache Spark and want to adopt dbt's transformation and testing practices. Ensure your Spark cluster is reachable and you have Python 3.10 or later.

## Install

pip install dbt-spark
uv add dbt-spark
poetry add dbt-spark

## Installing dbt-spark

Before you install: Low install friction with a pure-Python wheel. Actively maintained with a release 29 days ago and recent commits. Requires Python 3.10 or later and depends on dbt-core, dbt-adapters, dbt-common, and sqlparams.

License in practice: Licensed under Apache Software License (permissive), allowing commercial use, modification, and distribution with minimal restrictions.

Quickstart:

pip install dbt-spark

Create a dbt profile with Spark connection details (host, port, schema), then run:
dbt run

Requires Python 3.10 or later; a running Spark instance (local, Thrift server, or cloud) with network connectivity; Spark 3.3.2 or later supported.

Verify before relying:

- Whether all Spark connection methods (thrift, http, databricks, etc.) are equally well-supported in this version.
- Performance characteristics and scalability limits for large transformation workflows.
- Compatibility matrix with specific Spark and Hadoop versions beyond the stated 3.3.2 support.

## Package facts

- License: not declared (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 6.5M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

dbt spark adapter, transform data in spark, spark dbt integration, elt with apache spark, dbt warehouse adapter, spark sql transformation, data modeling spark, data-transformation, spark-integration, elt-pipeline

[View on SkillFed](https://skillfed.io/packages/dbt-spark) · [View on PyPI](https://pypi.org/project/dbt-spark/)