{"categories":[{"label":"Distributed Computing","url":"https://skillfed.io/packages/category/system-distributed-computing"}],"enrichment":{"capability":"Integrates Celery as a distributed task execution backend for Dagster data pipelines, enabling horizontal scaling of asset materialization and job runs across multiple worker nodes.","skillfed_tags":["distributed-execution","task-queue","orchestration"],"use_cases":["Scale Dagster asset materialization across multiple machines in production environments.","Isolate long-running or resource-intensive data transformations on dedicated worker nodes.","Integrate Dagster pipelines with existing Celery infrastructure and message brokers.","Enable parallel execution of independent Dagster tasks within a single job run.","Deploy Dagster in cloud-native environments where distributed execution is required."],"what_it_does":"dagster-celery is a Dagster integration that plugs Celery in as the execution engine for running Dagster assets and jobs. Instead of executing tasks sequentially or on a single machine, this package lets you distribute work across a cluster of Celery workers, enabling parallel execution and horizontal scaling of your data pipelines.\n\nThe package acts as a bridge between Dagster's declarative asset model and Celery's distributed task queue. When you configure a Dagster job or asset with the celery_executor, each task runs as a Celery job on an available worker node. This is useful for production deployments where you need to scale beyond a single machine or isolate compute-heavy workloads. The package is maintained as part of the main Dagster project and is actively developed.","worth_installing":"Yes, if you need to scale Dagster beyond a single machine or integrate with existing Celery infrastructure. The package is actively maintained, has no known vulnerabilities, and carries a permissive license. Install only if you have a Celery broker (RabbitMQ, Redis, etc.) already running or planned; it adds complexity that single-machine deployments do not need."},"id":"dagster-celery","links":{"html":"https://skillfed.io/packages/dagster-celery","md":"https://skillfed.io/packages/dagster-celery.md","pypi":"https://pypi.org/project/dagster-celery/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-08-14","license_spdx":"Apache-2.0","license_treatment":"permissive","name":"dagster-celery","python_support":"supports_current","summary":"Package for using Celery as Dagster's execution engine."},"popularity":{"monthly_downloads":3912612,"position":2454,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"0.29.18"}