dbt-glue

What it is and what it does

dbt-glue bridges dbt's transformation framework with AWS Glue's managed Spark engine, letting data teams write dbt models that execute as Spark jobs on Glue. It implements the dbt adapter protocol to translate dbt's SQL and Jinja templates into Spark-compatible operations, handling table creation, partitioning, and metadata management through Glue's catalog and Lake Formation permissions.

The adapter integrates with Glue Interactive Sessions, meaning your dbt runs spin up Spark clusters on demand rather than requiring persistent infrastructure. It depends on dbt-core, dbt-spark, boto3, and pyarrow, and requires AWS IAM roles and S3 bucket access for both metadata and data storage. Configuration happens through standard dbt profiles, specifying your Glue role, region, worker count, and S3 location for output data.

Use it for:

• Transform raw data in S3 using dbt models, with results catalogued in Glue and queryable via Athena or other tools.
• Build ELT pipelines that cleanse and denormalize data in place using Spark, coordinated through dbt's DAG and testing framework.
• Manage data lake transformations with Lake Formation permissions, using dbt to orchestrate schema and table updates.
• Run scheduled dbt jobs on AWS Glue without managing Spark clusters, leveraging Glue's serverless session provisioning.
• Integrate dbt workflows into AWS data platforms that already use Glue for ETL, unifying transformation logic.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

dbt-glue is a dbt adapter that enables running dbt transformations against AWS Glue's Spark engine using the Glue Interactive Sessions API.

Yes, if you are already running AWS Glue and want to adopt dbt for transformation orchestration and testing. The adapter is actively maintained, has low install friction, and permissive licensing. However, verify that your use case aligns with Spark's capabilities and Glue's cost model, and confirm production-readiness expectations given the Beta status.

Install

dbt-glue on PyPI

pip install dbt-glue

uv add dbt-glue

poetry add dbt-glue

Installing dbt-glue

Before you install

Low install friction with a pure Python wheel. Active maintenance with recent commits and a stable release cadence; marked as Beta status. Requires dbt-core, dbt-spark, and AWS SDK dependencies (boto3), all of which are well-maintained.

License in practice

Permissive license treatment means you can use this adapter in commercial and proprietary projects without significant legal constraints.

Quickstart

pip install dbt-glue

# In profiles.yml:
type: glue
role_arn: arn:aws:iam::ACCOUNT:role/GlueRole
region: us-east-1
schema: dbt_demo
location: s3://bucket/path

Requires Python 3.9 or later; AWS credentials and IAM permissions for Glue, Lake Formation, and S3 access must be configured; boto3 and AWS CLI setup recommended.

Verify before relying

• Whether the adapter supports all dbt features or has known limitations in Spark/Glue compatibility.
• Performance characteristics and cost implications of running dbt models through Glue Interactive Sessions.
• Maturity level beyond 'Beta' status and production-readiness guarantees.

Package facts

Evidence: dbt_glue-1.10.19-py3-none-any.whl

Tags

More Database packages