Packages
A Python client library for creating and managing isolated sandbox environments on Kubernetes clusters, supporting multiple connection modes from local development to production cloud deployments.
Install it if you are building agents or orchestration tools on Kubernetes and need reliable, scalable sandbox lifecycle management.
Integrates PySpark with Dagster's data pipeline orchestration, enabling you to define and run Spark-based data assets within Dagster's declarative asset model.
Converts Python cProfile statistics into interactive flamegraph visualizations, either as native SVG or as trace logs compatible with flamegraph.pl.
Integrates Snowflake with Dagster's data orchestration framework, enabling you to build and manage data pipelines that read from and write to Snowflake as part of a larger asset-driven workflow.
Install only if Dagster is already your orchestration framework; it is not a standalone Snowflake client.
A Python client library for running and managing isolated container sandboxes on CoreWeave infrastructure, with both sync and async APIs for executing commands and managing sandbox lifecycle.
Integrates Databricks with Dagster's data orchestration framework, enabling you to define and run data pipelines that interact with Databricks clusters and SQL warehouses.
Install only if you need Databricks-specific orchestration components; the base dagster library alone may suffice for simpler use cases.
Django Q2 is a multiprocessing distributed task queue for Django that lets you offload work to background workers, schedule recurring tasks, and monitor job execution through Django Admin.
Install it if you need background task processing and prefer a simpler, Django-native solution over a separate message broker architecture.
Dagster-spark integrates Apache Spark with Dagster's data orchestration framework, enabling you to define and run Spark-based data assets and pipelines within Dagster's declarative programming model.
Pottery provides Python dict, set, list, deque, queue, and counter interfaces backed by Redis, letting you use familiar Python collection syntax to read and write data persisted in Redis.
Provides a Python interface to NVSHMEM, enabling GPU-accelerated symmetric heap memory operations for inter-GPU communication on NVIDIA systems.
However, installation requires CUDA 13 and the nvidia-nvshmem-cu13 library to be pre-installed, and the unclear license status warrants verification before use in…
oslo.concurrency provides locking mechanisms and external process utilities for safely running multi-thread and multi-process applications.
Python bindings to the OpenStack Compute API, providing programmatic access to OpenStack Nova compute resources and a command-line interface for managing instances, images, and other compute operations.
Python bindings and command-line interface to the OpenStack Cinder block storage API, implementing 100% of the API surface for volume management, backups, snapshots, and related operations.
Install it if you need to manage OpenStack block storage from Python or the command line.
Provides distributed locking on top of DynamoDB with configurable retry semantics, heartbeat-based lease management, and lock-theft detection.
However, dormant maintenance (no updates since 2018-10-28) and classifiers limited to Python 3.6 and 3.7 raise questions about compatibility with modern Python versions.
Oslo Messaging provides a unified API for RPC and notification messaging across multiple transport backends, abstracting away the complexity of different messaging systems.
Futurist provides futures utilities and executors that add transparency and statistics gathering to asynchronous work execution, including eventlet and synchronous executor implementations.
Install only if your use case specifically requires the statistics gathering, eventlet executor, or synchronous executor—standard concurrent.futures covers most cases.
Dagit is the web UI for Dagster, providing a visual interface to orchestrate, monitor, and manage data pipelines and assets defined as Python functions.
Install it if you are already using Dagster or evaluating it for data pipeline orchestration; it is essential for visual monitoring and management of Dagster assets.
Distributes XGBoost training and inference across multiple nodes and GPUs using Ray as the underlying compute framework.
Provides a checkpoint store implementation for Azure Event Hubs consumers using Azure Blob Storage to persist consumer position and partition ownership data.
Install it if you are building an Event Hubs consumer application that requires durable, distributed checkpoint management.
A Flask-based web dashboard for monitoring RQ job queues, workers, and jobs in real time, deployable as a standalone service or integrated into an existing Flask application.
Dapr SDK for Python provides client libraries and actor framework support for building distributed applications using the Dapr runtime, with built-in support for gRPC, FastAPI, and Flask integrations.
However, it is a client library only—it requires a separate Dapr runtime to be useful, so evaluate whether Dapr's architecture and building blocks fit your…
Wraps Python's ProcessPoolExecutor and ThreadPoolExecutor to limit queued tasks, preventing memory exhaustion when submitting millions of jobs by only enqueuing new tasks as workers become available.
However, verify compatibility with your Python version first, as the latest release was 2019-06-04 and no updates are planned.
Provides Python client library for managing Azure Automation resources including runbooks, schedules, jobs, and hybrid workers via the Azure Resource Manager API.
Provides Python type definitions and model classes for Kubernetes resources, designed to work with the lightkube client library.
Python wrapper for Google's CityHash and FarmHash non-cryptographic hash functions, offering 32-, 64-, and 128-bit implementations with support for hardware-independent fingerprints and fast hashing of buffer-protocol objects like NumPy arrays.
A Python worker and client library for Faktory, a language-agnostic job server; lets you consume jobs from Faktory or push jobs to it for processing.
However, do not adopt it for new projects expecting active maintenance—the dormant status and lack of updates since 2023 mean you will be on your own for…
fal is a serverless Python runtime that packages and deploys your code to the cloud, handling infrastructure management and auto-scaling from zero when idle.
However, verify the license terms first (currently unclear in metadata) and confirm pricing and cold-start performance meet your requirements before committing to…
pqdm wraps tqdm and concurrent.futures to parallelize iteration over iterables with a live progress bar, supporting both process and thread-based parallelism.
However, if you require active maintenance, ongoing feature development, or support for Python versions beyond 3.9, consider whether a more actively maintained…
Durabletask is a Python SDK for building durable orchestrations and workflows that can survive failures and be replayed reliably, designed to work with Azure Durable Task Scheduler and Azure Durable Functions.
Faust-streaming is a Python library for building stream processing applications that consume from Kafka topics, perform stateful transformations, and persist distributed key-value state using RocksDB.
Not recommended if you need a simpler event queue or if your team lacks async/await experience.
aiometer is a concurrency scheduling library that limits how many async tasks run simultaneously and controls their rate, working with both asyncio and trio.
Generates backup, restore, and cleanup DAGs for Amazon MWAA metadata stores by providing factory classes that create reusable workflows without manual DAG authoring.
Provides a Python provider implementation for orchestrating durable workflows with Azure Durable Task Scheduler, enabling reliable task execution and state management across distributed systems.
Install only if you are committed to Azure Durable Task Scheduler or Azure Durable Functions; it is not a general-purpose workflow library.
Pyro5 enables remote method calls on Python objects across a network, letting you invoke methods on objects running on other machines as if they were local.
Pulumi component for provisioning and managing AWS EKS Kubernetes clusters, including control plane, worker node groups, and AWS CNI networking.
Install it if you are using Pulumi to manage AWS infrastructure and need to provision EKS clusters; it significantly reduces boilerplate compared to managing raw AWS…
Integrates dlt data loading into Dagster pipelines, enabling declarative ETL/ELT workflows where dlt sources and destinations are orchestrated as Dagster assets.
Install it if you are already using both Dagster and dlt and want to orchestrate dlt jobs as first-class Dagster assets rather than external processes.
Prevents duplicate Celery tasks from being queued or running simultaneously by using task name and arguments to determine uniqueness, with Redis-backed distributed locking.
However, proceed with caution: the package is abandoned (last release 2021-01-14, last commit 2023-06-09), so compatibility with recent Celery versions is unverified…
Provides Python API bindings for Argo Workflows, enabling programmatic submission and management of container-native workflow jobs on Kubernetes clusters.
However, verify the license terms before use—the metadata shows the license as unclear, which is a blocker for some organizations.
Distributed image processing using Dask, enabling large-scale image operations across multiple cores or machines by parallelizing computations that would otherwise be memory-bound on a single system.
However, Pre-Alpha status means the API may change and feature coverage is incomplete—evaluate whether the available operations match your use case before committing…
Coordinates communication between multiple AI agents running in separate OpenCode sessions, enabling them to discover each other and exchange messages through a daemon.
However, be aware that it is still in Beta, the project is young, and injected messages are not visible in the OpenCode TUI, which may complicate debugging.