{"categories":[{"label":"Distributed Computing","url":"https://skillfed.io/packages/category/system-distributed-computing"}],"enrichment":{"capability":"Integrates Dagster data orchestration with Celery task queue and Kubernetes execution, enabling distributed pipeline runs across containerized infrastructure.","skillfed_tags":["orchestration","kubernetes","celery-integration"],"use_cases":["Running Dagster data pipelines at scale across a Kubernetes cluster with multiple worker nodes and dynamic pod scheduling.","Isolating pipeline workloads in a multi-tenant Dagster deployment where different teams or projects need separate compute resources.","Distributing long-running asset computations (ETL, model training, report generation) across containerized workers to reduce latency.","Integrating Dagster with existing Celery infrastructure and Kubernetes deployments already in place at your organization.","Enabling CI/CD best practices by running asset definitions and tests in ephemeral Kubernetes pods provisioned on demand."],"what_it_does":"dagster-celery-k8s is an integration package that bridges Dagster's data orchestration framework with Celery task queue execution on Kubernetes. It provides an executor that allows Dagster jobs to distribute individual asset computations and task runs across a Kubernetes cluster using Celery as the task queue backend, enabling horizontal scaling and multi-tenant isolation.\n\nThe package is part of Dagster's ecosystem and is used when you want to run Dagster pipelines in production on Kubernetes infrastructure. Rather than running all tasks in a single process or on a single machine, this executor queues tasks to Celery workers deployed as Kubernetes pods, allowing you to scale compute independently and run many pipelines concurrently across your cluster. It is typically installed alongside dagster, dagster-celery, and dagster-k8s as part of a larger orchestration deployment.","worth_installing":"Yes, if you are running Dagster in production on Kubernetes and need distributed, scalable task execution. Install it as part of your Dagster orchestration stack when you have Celery and Kubernetes infrastructure already in place. Not needed for local development or single-machine deployments. No known security vulnerabilities and actively maintained."},"id":"dagster-celery-k8s","links":{"html":"https://skillfed.io/packages/dagster-celery-k8s","md":"https://skillfed.io/packages/dagster-celery-k8s.md","pypi":"https://pypi.org/project/dagster-celery-k8s/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-08-14","license_spdx":"Apache-2.0","license_treatment":"permissive","name":"dagster-celery-k8s","python_support":"supports_current","summary":"A Dagster integration for celery-k8s-executor"},"popularity":{"monthly_downloads":1920540,"position":3430,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"0.29.18"}