Packages
Provides Azure-specific integrations for Dagster, enabling data pipelines to interact with Azure services like Blob Storage, Data Lake, and Machine Learning.
Provides shell operation support for Dagster data pipelines, enabling you to execute shell commands and scripts as part of your orchestrated data workflows.
Provides Python type models for Kubernetes resources, generated from OpenAPI specifications, enabling type-safe interaction with Kubernetes APIs.
Not recommended if you need active maintenance or support for the latest Kubernetes features.
cwltool is the reference implementation of the Common Workflow Language standard, enabling you to write, validate, and execute portable scientific workflows defined in CWL format across different computing environments.
Install it if you need to execute portable workflows, validate CWL definitions, or integrate CWL workflows into Python applications.
A Python client for Apache Livy that enables remote code execution on Spark clusters via Livy's REST interface.
H2O Wave is a Python framework for building interactive web applications with real-time state synchronization, form components, and data visualization—no HTML, JavaScript, or CSS required.
Prevents duplicate execution and queuing of Celery tasks by enforcing distributed locks, allowing you to ensure a task runs only once even if scheduled multiple times concurrently.
Mooncake Transfer Engine provides high-performance, zero-copy data transfer for distributed KV cache management in large-scale LLM inference and training, using RDMA and P2P mechanisms to move cache and model weights across GPU clusters.
Atomic-Dict provides a lock-free, shared 64-bit integer key-value map for multiprocess synchronization using atomic operations like compare-and-swap, exchange, and increment.
Extends Kubeflow Pipelines SDK with Kubernetes-native features like Secrets, PersistentVolumeClaims, node selectors, tolerations, and pod metadata configuration for pipeline tasks.
PgQueuer turns PostgreSQL into a background job queue, letting you enqueue, schedule, and process jobs transactionally within your existing database instead of running a separate message broker.
ClusterShell is a Python framework for running commands in parallel across many nodes in a cluster, with tools to manage node groups, aggregate results, and handle distributed execution.
Install it if you manage or script operations on Linux clusters or HPC systems and need parallel command execution with node group abstractions.
A Python client library for etcd, a distributed key-value store, enabling read, write, delete, and watch operations on etcd servers.
Integrates DuckDB with Dagster's data pipeline orchestration, providing DuckDB-specific ops and resources for declaring and running data assets.
Install it if you are already using Dagster and want to work with DuckDB; it adds minimal overhead and is backed by a well-resourced open-source project.
DrJAX embeds MapReduce programming primitives into JAX, enabling differentiable distributed and parallel computations with optimized execution across large-scale datacenter settings.
However, it remains early-stage (first release January 2025) with modest adoption; verify that its API and performance characteristics match your production…
pykube-ng is a Python client library for querying, creating, and managing Kubernetes API objects through a native Python interface backed by requests and kubeconfig authentication.
However, do not adopt it for new projects: the package is abandoned (no releases in 1155+ days), so it will not receive security patches or updates for newer…
Pins compatible versions of dask and distributed for RAPIDS packages, and patches dask modules on import to ensure RAPIDS compatibility.
Python client library for querying and managing Apache Hadoop YARN clusters via REST API, supporting resource managers, application masters, history servers, and node managers.
However, do not install if you require active maintenance, bug fixes, or compatibility with newer Hadoop versions—the project is abandoned and has not been updated…
Python client for etcd v3 API that provides key-value operations, distributed locks, transactions, and key watching via gRPC.
Implements Redis-based distributed locks using the Redlock algorithm, allowing you to acquire and release locks across multiple Redis instances for coordinating access to shared resources.
Routes Django email sending through Celery task queues for asynchronous, out-of-band message delivery instead of blocking the request.
However, proceed with caution: the repository is archived and unmaintained since 2019, so compatibility with modern Django (4.0+) and Celery (5.x+) is unverified.
A Python library providing asynchronous bindings to the Juju orchestration platform, enabling programmatic deployment and management of applications and infrastructure through an object-oriented API.
However, the repository is archived and abandoned, with no commits since 2026-07-22 and no releases in 399 days.
Lithops is a Python framework for running unmodified code at scale across cloud providers, HPC, and on-premise platforms using serverless compute and storage backends.
The main gotcha is that you must configure a compute and storage backend before running anything; it is not a drop-in replacement for local parallelism without setup.
Loads, parses, and manipulates Common Workflow Language (CWL) documents across versions 1.0, 1.1, and 1.2, providing autogenerated typed classes and CLI utilities for workflow introspection and transformation.
SAQ is an async job queue framework that runs background tasks using asyncio, backed by Redis or PostgreSQL, with optional cron scheduling and a web monitoring UI.
Python client library and command-line tool for interacting with OpenStack Heat orchestration services, enabling programmatic stack management and template deployment.
Provides an executable hook that fetches HTCondor configuration files from a git repository and dynamically includes them in HTCondor config files via the `include command` syntax.
Python API for Apache Flink that builds stateful batch and streaming data processing pipelines, offering both high-level Table API (SQL-like queries) and lower-level DataStream API for complex stream processing.
Not recommended for lightweight, single-machine workloads or if you want to avoid Java runtime overhead.
Integrates Snowflake and Pandas with Dagster's data pipeline orchestration, enabling you to define, run, and observe data assets built on Snowflake and processed with Pandas.
A command-line tool and Python client for managing Apache Kafka Connect clusters and connectors via the Kafka Connect REST API, supporting connector creation, configuration, status inspection, and lifecycle operations.
However, the aging maintenance status warrants caution for production use—verify that the package still covers your Kafka Connect version's API surface and consider…
Builds long-running AWS Lambda workflows with automatic checkpointing, durable steps, waits, callbacks, and parallel execution that can resume after Lambda pauses or restarts.
Integrates Airbyte data connectors with Dagster's orchestration engine to define and run data ingestion assets as part of declarative data pipelines.
Install it if you are already using Dagster for orchestration and want to integrate Airbyte connectors as declarative assets; it is not necessary if you are not using…
IxNetwork is a Python client library for configuring and running tests against Ixia's IxNetwork API server, supporting both Windows and Linux (Web Edition) deployments.
However, the 966-day gap since the last release and dormant maintenance status mean you should verify compatibility with your specific IxNetwork server version and…
Bernstein orchestrates parallel CLI coding agents (Claude Code, Codex, Gemini CLI, and 40+ others) with deterministic replay, cryptographic lineage tracking, and optional audit chains—no LLM in the coordination loop, so runs are reproducible end-to-end.
ZenML is an MLOps orchestration platform that lets you write pipelines and deploy them across any infrastructure backend, automatically handling containerization, run tracking, and integration with existing ML tools.
Cotyledon provides a framework for building long-running services with Unix signal handling, worker process spawning, supervision, daemon reloading, and rate-limited worker lifecycle management.
Install it if you are building a service that spawns multiple workers and want clean, predictable process management.
Celery Batches provides a Task class that buffers multiple Celery task calls and processes them together as a list, flushing on a timer or when a size threshold is reached.
Provides a Python guest runtime for the Hyperlight Wasm sandbox backend, enabling Python code execution within isolated Wasm environments.
However, it is only useful as part of a Hyperlight deployment—verify that it integrates with your version of the main Hyperlight package before installing.
PyPIM provides a Python client to communicate with the Ansys Product Instance Management (PIM) API, a gRPC service for starting and managing remote product instances.
However, the PIM service itself is not publicly exposed, so this is primarily useful within Ansys infrastructure or authorized environments.
Manages clusters of IPython processes for parallel computing, providing CLI tools and a Python API to distribute computations across multiple engines via the Jupyter protocol.