dbt-spark
The Apache Spark adapter plugin for dbt
Decision gist · record as of 2026-08-14
Yes. dbt-spark is production-ready, actively maintained, has no known vulnerabilities, low install friction, and a permissive license. Install it if you use Apache Spark and want to adopt dbt's transformation and testing practices. Ensure your Spark cluster is reachable and you have Python 3.10 or later.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.10 or later; a running Spark instance (local, Thrift server, or cloud) with network connectivity; Spark 3.3.2 or later supported.
- Low install friction with a pure-Python wheel.
- Actively maintained with a release 29 days ago and recent commits.
License · maintenance · safety
permissive license (permissive) — Licensed under Apache Software License (permissive), allowing commercial use, modification, and distribution with minimal restrictions.
last release 2026-07-16 (29 days) · last repo commit 2026-08-14 · 231 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 6,546,930 downloads/mo, #1,891 on PyPI
Alternatives
Verify before relying
pip install dbt-spark
Create a dbt profile with Spark connection details (host, port, schema), then run:
dbt run- Whether all Spark connection methods (thrift, http, databricks, etc.) are equally well-supported in this version.
- Performance characteristics and scalability limits for large transformation workflows.
- Compatibility matrix with specific Spark and Hadoop versions beyond the stated 3.3.2 support.
What it is and what it does
dbt-spark is a plugin that connects dbt (a data transformation tool built on SQL and YAML) to Apache Spark, allowing data engineers and analysts to apply software engineering practices—version control, testing, documentation, modularity—to Spark-based data pipelines. It handles the translation of dbt's declarative transformation models into Spark SQL, managing the execution and metadata tracking within a Spark cluster or local Thrift server.
The adapter sits between dbt-core and Spark, inheriting dbt's ecosystem (dbt-adapters, dbt-common) while adding Spark-specific connection logic and SQL dialect support. It is production-ready (Development Status 5), actively maintained, and supports modern Python versions (3.10–3.13) across Linux, macOS, and Windows.
Use it for
- Transform raw data in a Spark warehouse using dbt models, organizing and cleansing data for downstream analytics.
- Build modular, testable data pipelines in Spark with version control and documentation via dbt's YAML and SQL interface.
- Integrate Spark-based ELT workflows into a dbt Cloud or dbt Core project alongside other warehouse adapters.
- Run local Spark development and testing using the docker-compose Thrift server setup for rapid iteration.
- Manage Hive Metastore schemas and table lineage through dbt's metadata layer on top of Spark.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
dbt-spark is production-ready, actively maintained, has no known vulnerabilities, low install friction, and a permissive license. Install it if you use Apache Spark and want to adopt dbt's transformation and testing practices. Ensure your Spark cluster is reachable and you have Python 3.10 or later.
Install
dbt-spark on PyPI
Before you install
Low install friction with a pure-Python wheel. Actively maintained with a release 29 days ago and recent commits. Requires Python 3.10 or later and depends on dbt-core, dbt-adapters, dbt-common, and sqlparams.
Requires Python 3.10 or later; a running Spark instance (local, Thrift server, or cloud) with network connectivity; Spark 3.3.2 or later supported.
License in practice
Licensed under Apache Software License (permissive), allowing commercial use, modification, and distribution with minimal restrictions.
Quickstart
pip install dbt-spark
Create a dbt profile with Spark connection details (host, port, schema), then run:
dbt runVerify before relying
- Whether all Spark connection methods (thrift, http, databricks, etc.) are equally well-supported in this version.
- Performance characteristics and scalability limits for large transformation workflows.
- Compatibility matrix with specific Spark and Hadoop versions beyond the stated 3.3.2 support.
Package facts
| License | permissive license permissive |
| Python support | Supports the current Python release >=3.10.0 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 4 packagesdbt-adaptersdbt-commondbt-coresqlparams |
| Maintenance | Actively maintained 29 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 6,546,930 / month, #1,891 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 5 - Production/StableLicense :: OSI Approved :: Apache Software LicenseOperating System :: MacOS :: MacOS XOperating System :: Microsoft :: WindowsOperating System :: POSIX :: LinuxProgramming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13 |
Evidence: dbt_spark-1.11.0-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “dbt spark adapter”
- dbt-sparkdbt-spark is the Apache Spark adapter for dbt, enabling data…
- dbt-gluedbt-glue is a dbt adapter that enables running dbt transformations…
- dbt-fabricsparkA dbt adapter that connects dbt to Apache Spark in Microsoft Fabric…
Give your agent the search over MCP, or paste the wish link into any chat.
More Database packages
psycopg2-binary is a PostgreSQL database adapter for Python that implements the DB API 2.0 specification, enabling Python applications to connect to and query PostgreSQL databases with thread-safe concurrent operations.
Python client library for connecting to and executing commands against Redis key-value stores, supporting both synchronous and asynchronous operations.
Install it if your application needs to interact with Redis; the only prerequisite is a running Redis server instance.
YDB Python SDK is the official client library for connecting to and querying YDB databases from Python applications.
Install it if you need to connect Python applications to YDB databases.
Connects Python applications to Snowflake data warehouses using the DB API 2.0 specification, enabling SQL queries, data transfers, and warehouse operations.
sqlparse tokenizes SQL text into a tree of statements, clauses, and expressions, and provides functions to split scripts, format queries, and inspect parsed tokens without validating dialect or syntax.
Install it if you need to manipulate, format, or analyze SQL text programmatically.
Provides base adapter protocols and shared functionality that database adapters use to integrate with dbt-core, handling connections, dialect translation, relation caching, and core interface management.
See also dbt-bigquery · dbt-databricks · dbt-snowflake · dbt-redshift · dbt-oracle · dbt-postgres · dbt-fabricspark · dbt-glue · dbt-core · dbt-athena-community