dagster-shell

What it is and what it does

dagster-shell is a library that extends Dagster with the ability to define and orchestrate shell operations within your data pipelines. It wraps shell command execution as first-class Dagster ops, allowing you to integrate system-level scripts, command-line tools, and shell workflows into your declarative asset and job definitions. This is useful when you have existing shell scripts, need to invoke system utilities, or want to orchestrate heterogeneous tools that expose command-line interfaces.

The package is part of the Dagster ecosystem and inherits its design philosophy: your shell operations are defined as Python functions decorated with Dagster markers, then scheduled, monitored, and lineage-tracked through Dagster's unified control plane. It supports Python 3.9 through 3.12 and is actively maintained by the Dagster team.

Use it for:

• Invoke existing shell scripts or command-line tools as part of a larger Dagster pipeline without rewriting them in Python.
• Execute system administration tasks like file transfers, backups, or environment setup as orchestrated pipeline steps.
• Integrate third-party CLI tools (e.g., dbt, terraform, custom binaries) into a Dagster-managed data workflow.
• Run data processing commands that are simpler or more efficient to express in shell than in Python code.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides shell operation support for Dagster data pipelines, enabling you to execute shell commands and scripts as part of your orchestrated data workflows.

Yes, if you have existing shell scripts or command-line tools that need to be part of a Dagster pipeline. The package is lightweight, actively maintained, has no known vulnerabilities, and integrates seamlessly with Dagster's orchestration model. Install it only if you actually need shell operation support; if your pipeline is purely Python-based, it adds no value.

Install

dagster-shell on PyPI

pip install dagster-shell

uv add dagster-shell

poetry add dagster-shell

Installing dagster-shell

Before you install

Low friction install with a single runtime dependency on dagster. The package is actively maintained with recent releases and no known vulnerabilities.

License in practice

Apache 2.0 licensed under a permissive regime, allowing commercial use and modification with minimal restrictions.

Quickstart

pip install dagster-shell

import dagster as dg
from dagster_shell import shell_op

@shell_op
def my_shell_task():
    return "echo 'Hello from shell'"

Requires dagster to be installed; shell commands execute in the system environment where Dagster runs.

Verify before relying

• Whether shell_op supports environment variable injection and credential handling for sensitive operations.
• How error handling and exit codes from shell commands are surfaced in Dagster's observability layer.
• Whether the package provides built-in support for common shell patterns like piping, redirection, or conditional execution.

Package facts

Evidence: dagster_shell-0.25.13-py3-none-any.whl

Tags

More Distributed Computing packages