{"categories":[{"label":"Distributed Computing","url":"https://skillfed.io/packages/category/system-distributed-computing"}],"enrichment":{"capability":"Dagster-pipes provides a toolkit for running Dagster integrations and transform logic outside of the main Dagster process, enabling external execution of data pipelines.","skillfed_tags":["orchestration","data-pipelines","external-execution"],"use_cases":["Execute dbt models or SQL transforms in a remote database while orchestrating them through Dagster.","Run Python transforms in isolated containers or Kubernetes pods with Dagster managing the workflow.","Integrate third-party tools or legacy systems that cannot run directly in the Dagster process.","Scale compute-heavy transformations to specialized infrastructure while keeping Dagster as the control plane.","Build multi-environment pipelines where different stages run on different systems or cloud providers."],"what_it_does":"Dagster-pipes is a specialized toolkit within the Dagster ecosystem that enables you to run transform logic and integrations outside the main Dagster process. It sits between Dagster's orchestration engine and external compute environments, allowing you to execute data transformations remotely while maintaining lineage and observability through Dagster's central control plane.\n\nThe package is designed for scenarios where your data processing needs to run in a separate environment\u2014such as a different container, machine, or cloud service\u2014but you still want Dagster to coordinate, monitor, and track the results. It abstracts away the complexity of subprocess communication and result serialization, letting you focus on defining what work happens where rather than managing the plumbing.","worth_installing":"Yes, if you are already using Dagster and need to execute transforms or integrations outside the main process. The active maintenance, zero external dependencies, and permissive license make it a low-friction addition to a Dagster deployment. Not relevant for non-Dagster workflows."},"id":"dagster-pipes","links":{"html":"https://skillfed.io/packages/dagster-pipes","md":"https://skillfed.io/packages/dagster-pipes.md","pypi":"https://pypi.org/project/dagster-pipes/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-08-07","license_spdx":"Apache-2.0","license_treatment":"permissive","name":"dagster-pipes","python_support":"supports_current","summary":"Toolkit for Dagster integrations with transform logic outside of Dagster"},"popularity":{"monthly_downloads":9387238,"position":1538,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"1.13.17"}