dagster-gcp

What it is and what it does

dagster-gcp is a library that extends Dagster's orchestration capabilities with first-class support for Google Cloud Platform services. It provides resources and integrations for BigQuery, Cloud Storage, and other GCP tools, allowing you to declare data assets that live in or depend on GCP infrastructure and have Dagster manage their execution and lineage.

The package sits between Dagster's core orchestration engine and the GCP Python client libraries (google-cloud-bigquery, google-cloud-storage, google-api-python-client). It abstracts away credential management and connection pooling, letting you focus on defining what data assets you need rather than how to connect to them. It is part of the broader Dagster ecosystem and shares the same declarative, asset-oriented programming model.

Use it for:

• Build and orchestrate data transformation pipelines that read from or write to BigQuery tables
• Manage machine learning model training workflows that use GCP compute and store results in Cloud Storage
• Centralize lineage and observability for data assets distributed across multiple GCP projects
• Integrate GCP data sources into a multi-cloud or hybrid data platform using Dagster as the control plane

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides GCP-specific integrations for Dagster, enabling data pipelines to interact with Google Cloud services like BigQuery and Cloud Storage.

Yes, if you are already using Dagster and need to orchestrate work on GCP. The package is actively maintained, has no known vulnerabilities, and low install friction. It is a natural extension of Dagster for teams committed to that orchestration model. If you are not yet using Dagster, evaluate the core platform first—this is an add-on, not a standalone tool.

Install

dagster-gcp on PyPI

pip install dagster-gcp

uv add dagster-gcp

poetry add dagster-gcp

Installing dagster-gcp

Before you install

Low install friction with a wheel distribution. Actively maintained with recent releases. Depends on core Dagster plus GCP client libraries, all standard packages.

License in practice

Apache 2.0 licensed, permissive terms allow commercial and private use with minimal restrictions.

Quickstart

pip install dagster-gcp

import dagster as dg
from dagster_gcp import BigQueryResource

@dg.asset
def my_asset(bigquery: BigQueryResource):
    return bigquery.get_client().query("SELECT").result()

Requires Python 3.10 or later (up to 3.14). GCP credentials must be configured in your environment.

Verify before relying

• Specific BigQuery dataset/table operations supported beyond basic client access
• Whether Cloud Storage integration includes direct read/write asset I/O or only resource provisioning
• Performance characteristics when orchestrating large-scale GCP workloads

Package facts

Evidence: dagster_gcp-0.29.18-py3-none-any.whl

Tags

More Distributed Computing packages