apache-airflow-providers-celery

What it is and what it does

This is an Apache Airflow provider package that plugs Celery into Airflow as a distributed task executor. Instead of running all DAG tasks on a single Airflow scheduler, it routes them to remote Celery workers across a cluster, letting you scale horizontally by adding more worker nodes. The package wraps Celery's task queue mechanics and integrates them with Airflow's DAG scheduling, dependency tracking, and monitoring.

You install it alongside an existing Airflow setup, configure it to point to a Celery broker (typically Redis or RabbitMQ), and then set Airflow to use CeleryExecutor. Airflow will then push tasks to the broker, workers pull and execute them, and results flow back. The package includes integration with flower, Celery's monitoring tool, so you can observe worker status and task progress from a web UI.

Use it for:

• Scale Airflow task execution across multiple machines by running workers on separate nodes and routing DAG tasks through Celery.
• Process long-running or CPU-intensive tasks in parallel without blocking the Airflow scheduler.
• Monitor distributed task execution and worker health using flower's web dashboard.
• Run Airflow in a multi-tenant environment where different teams or projects need isolated worker pools.
• Integrate Airflow with an existing Celery infrastructure already in use for other applications.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Integrates Celery distributed task execution with Apache Airflow, enabling workers to process DAG tasks across a cluster using Celery as the task queue backend.

Yes, if you are running Apache Airflow and need to scale task execution beyond a single machine. The package is actively maintained, has no known vulnerabilities, and is production-stable. Install only if you have a Celery broker already running or planned; it is not useful without one. The low install friction and permissive Apache-2.0 license make it a straightforward addition to an existing Airflow deployment.

Install

apache-airflow-providers-celery on PyPI

pip install apache-airflow-providers-celery

uv add apache-airflow-providers-celery

poetry add apache-airflow-providers-celery

Installing apache-airflow-providers-celery

Before you install

Low friction install as a pure Python wheel. Actively maintained with a release 6 days old. Requires apache-airflow >=2.11.0, apache-airflow-providers-common-compat >=1.15.0, celery[redis] >=5.5.0,<6, and flower >=1.0.0.

License in practice

Apache-2.0 permissive license allows commercial and private use without restriction, typical for Apache Foundation projects.

Quickstart

pip install apache-airflow-providers-celery

from airflow.providers.celery.executors.celery_executor import CeleryExecutor

# Configure in airflow.cfg or environment
# executor = CeleryExecutor
# celery_broker_url = redis://localhost/0

Requires an existing Airflow installation (>=2.11.0) and a Celery broker (e.g., Redis or RabbitMQ) to be running separately.

Verify before relying

• Whether this provider supports Kubernetes pod operators or only traditional Celery workers
• Performance characteristics and scaling limits for typical cluster sizes
• Specific monitoring capabilities beyond what flower provides

Package facts

Evidence: apache_airflow_providers_celery-3.23.1-py3-none-any.whl

Keywords: airflow-provider, celery, airflow, integration

Tags

More Monitoring packages