---
id: dagster-celery
version: "0.29.18"
license: Apache-2.0
license_treatment: permissive
maintenance: active
---

# dagster-celery — Package for using Celery as Dagster's execution engine.

License: permissive · Maintenance: active · Downloads: 3.9M/mo

## What it is and what it does

dagster-celery is a Dagster integration that plugs Celery in as the execution engine for running Dagster assets and jobs. Instead of executing tasks sequentially or on a single machine, this package lets you distribute work across a cluster of Celery workers, enabling parallel execution and horizontal scaling of your data pipelines.

The package acts as a bridge between Dagster's declarative asset model and Celery's distributed task queue. When you configure a Dagster job or asset with the celery_executor, each task runs as a Celery job on an available worker node. This is useful for production deployments where you need to scale beyond a single machine or isolate compute-heavy workloads. The package is maintained as part of the main Dagster project and is actively developed.

Use it for:

- Scale Dagster asset materialization across multiple machines in production environments.
- Isolate long-running or resource-intensive data transformations on dedicated worker nodes.
- Integrate Dagster pipelines with existing Celery infrastructure and message brokers.
- Enable parallel execution of independent Dagster tasks within a single job run.
- Deploy Dagster in cloud-native environments where distributed execution is required.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Integrates Celery as a distributed task execution backend for Dagster data pipelines, enabling horizontal scaling of asset materialization and job runs across multiple worker nodes.

Yes, if you need to scale Dagster beyond a single machine or integrate with existing Celery infrastructure. The package is actively maintained, has no known vulnerabilities, and carries a permissive license. Install only if you have a Celery broker (RabbitMQ, Redis, etc.) already running or planned; it adds complexity that single-machine deployments do not need.

## Install

pip install dagster-celery
uv add dagster-celery
poetry add dagster-celery

## Installing dagster-celery

Before you install: Low friction installation with a pure-Python wheel and four runtime dependencies (celery, click, dagster, importlib-metadata). The package is actively maintained with a release on 2026-08-14 and no known vulnerabilities.

License in practice: Apache-2.0 permissive license allows use in commercial and private projects without copyleft obligations, making it suitable for enterprise deployments.

Quickstart:

pip install dagster-celery

import dagster as dg
from dagster_celery import celery_executor

@dg.job(executor_def=celery_executor)
def my_job():
    pass

Requires Celery broker (e.g. RabbitMQ, Redis) and worker processes running separately; Dagster must be Python 3.10 or later.

Verify before relying:

- Whether the package includes built-in configuration helpers or requires manual Celery broker setup.
- Performance characteristics and recommended worker count for typical workloads.
- Support for Celery task routing, priority queues, or other advanced Celery features.

## Package facts

- License: Apache-2.0 (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 3.9M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

celery executor for dagster, distributed task execution dagster, dagster celery integration, scale dagster jobs across workers, celery-based pipeline orchestration, dagster distributed execution, multi-worker dagster deployment, distributed-execution, task-queue, orchestration

[View on SkillFed](https://skillfed.io/packages/dagster-celery) · [View on PyPI](https://pypi.org/project/dagster-celery/)