dagster-celery

What it is and what it does

dagster-celery is a Dagster integration that plugs Celery in as the execution engine for running Dagster assets and jobs. Instead of executing tasks sequentially or on a single machine, this package lets you distribute work across a cluster of Celery workers, enabling parallel execution and horizontal scaling of your data pipelines.

The package acts as a bridge between Dagster's declarative asset model and Celery's distributed task queue. When you configure a Dagster job or asset with the celery_executor, each task runs as a Celery job on an available worker node. This is useful for production deployments where you need to scale beyond a single machine or isolate compute-heavy workloads. The package is maintained as part of the main Dagster project and is actively developed.

Use it for:

• Scale Dagster asset materialization across multiple machines in production environments.
• Isolate long-running or resource-intensive data transformations on dedicated worker nodes.
• Integrate Dagster pipelines with existing Celery infrastructure and message brokers.
• Enable parallel execution of independent Dagster tasks within a single job run.
• Deploy Dagster in cloud-native environments where distributed execution is required.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Integrates Celery as a distributed task execution backend for Dagster data pipelines, enabling horizontal scaling of asset materialization and job runs across multiple worker nodes.

Yes, if you need to scale Dagster beyond a single machine or integrate with existing Celery infrastructure. The package is actively maintained, has no known vulnerabilities, and carries a permissive license. Install only if you have a Celery broker (RabbitMQ, Redis, etc.) already running or planned; it adds complexity that single-machine deployments do not need.

Install

dagster-celery on PyPI

pip install dagster-celery

uv add dagster-celery

poetry add dagster-celery

Installing dagster-celery

Before you install

Low friction installation with a pure-Python wheel and four runtime dependencies (celery, click, dagster, importlib-metadata). The package is actively maintained with a release on 2026-08-14 and no known vulnerabilities.

License in practice

Apache-2.0 permissive license allows use in commercial and private projects without copyleft obligations, making it suitable for enterprise deployments.

Quickstart

pip install dagster-celery

import dagster as dg
from dagster_celery import celery_executor

@dg.job(executor_def=celery_executor)
def my_job():
    pass

Requires Celery broker (e.g. RabbitMQ, Redis) and worker processes running separately; Dagster must be Python 3.10 or later.

Verify before relying

• Whether the package includes built-in configuration helpers or requires manual Celery broker setup.
• Performance characteristics and recommended worker count for typical workloads.
• Support for Celery task routing, priority queues, or other advanced Celery features.

Package facts

Evidence: dagster_celery-0.29.18-py3-none-any.whl

Tags

More Distributed Computing packages