dagster-celery-k8s

What it is and what it does

dagster-celery-k8s is an integration package that bridges Dagster's data orchestration framework with Celery task queue execution on Kubernetes. It provides an executor that allows Dagster jobs to distribute individual asset computations and task runs across a Kubernetes cluster using Celery as the task queue backend, enabling horizontal scaling and multi-tenant isolation.

The package is part of Dagster's ecosystem and is used when you want to run Dagster pipelines in production on Kubernetes infrastructure. Rather than running all tasks in a single process or on a single machine, this executor queues tasks to Celery workers deployed as Kubernetes pods, allowing you to scale compute independently and run many pipelines concurrently across your cluster. It is typically installed alongside dagster, dagster-celery, and dagster-k8s as part of a larger orchestration deployment.

Use it for:

• Running Dagster data pipelines at scale across a Kubernetes cluster with multiple worker nodes and dynamic pod scheduling.
• Isolating pipeline workloads in a multi-tenant Dagster deployment where different teams or projects need separate compute resources.
• Distributing long-running asset computations (ETL, model training, report generation) across containerized workers to reduce latency.
• Integrating Dagster with existing Celery infrastructure and Kubernetes deployments already in place at your organization.
• Enabling CI/CD best practices by running asset definitions and tests in ephemeral Kubernetes pods provisioned on demand.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Integrates Dagster data orchestration with Celery task queue and Kubernetes execution, enabling distributed pipeline runs across containerized infrastructure.

Yes, if you are running Dagster in production on Kubernetes and need distributed, scalable task execution. Install it as part of your Dagster orchestration stack when you have Celery and Kubernetes infrastructure already in place. Not needed for local development or single-machine deployments. No known security vulnerabilities and actively maintained.

Install

dagster-celery-k8s on PyPI

pip install dagster-celery-k8s

uv add dagster-celery-k8s

poetry add dagster-celery-k8s

Installing dagster-celery-k8s

Before you install

Active maintenance with a release on 2026-08-14 and low install friction. Depends on dagster, dagster-celery, and dagster-k8s, all of which are core Dagster components. Suitable for production use in environments already running Dagster.

License in practice

Apache-2.0 licensed under permissive terms, allowing commercial use, modification, and distribution with minimal restrictions.

Quickstart

pip install dagster-celery-k8s

from dagster_celery_k8s import celery_k8s_job_executor
from dagster import job

@job(executor_def=celery_k8s_job_executor)
def my_pipeline():
    pass

Requires Python 3.10 or later (supports up to 3.14). Assumes Dagster, Celery, and Kubernetes infrastructure are already configured and running.

Verify before relying

• Whether this package is intended as a standalone installation or primarily as a dependency of dagster-k8s or dagster-celery.
• Performance characteristics and scaling limits when running large numbers of concurrent tasks across Kubernetes nodes.
• Whether additional configuration beyond the executor definition is required to connect to an existing Celery broker and Kubernetes cluster.

Package facts

Evidence: dagster_celery_k8s-0.29.18-py3-none-any.whl

Tags

More Distributed Computing packages