Packages
Python SDK for VMware vSphere Management API that enables programmatic control of ESXi and vCenter Server infrastructure.
Install it if you need to automate or integrate with VMware vSphere environments.
Client library for programmatically managing Azure Service Fabric clusters, applications, and services through REST APIs.
However, the last release was in 2021-12-14; verify that its Python version support and API coverage meet your current needs before committing to a dependency on it.
SkyPilot is a control plane for launching, managing, and scaling AI workloads across multiple cloud providers, Kubernetes clusters, Slurm, and on-premises infrastructure from a single unified interface.
Submitit wraps Slurm job submission and monitoring, letting you submit Python functions to a cluster and retrieve results, logs, and job status from within Python code.
Manages Azure Machine Learning Compute resources via the Azure Resource Manager API, providing Python bindings to create, configure, and delete compute clusters and related infrastructure.
However, verify that it aligns with your Azure SDK strategy—Microsoft has released newer, unified SDKs since 2018, so check whether your project should migrate to a…
Provides Python client library bindings for Azure Dev Spaces resource management through the Azure Resource Manager API.
However, note that the package has not been updated since 2019-05-23 and may lack support for newer Azure or Python features.
Provides utility functions and Jupyter integration helpers for the Sparkmagic project, enabling interactive Spark cluster access through Jupyter notebooks via REST-based Livy, Lighter, or Ilum servers.
Dramatiq is a distributed task processing library that lets you define Python functions as background workers and queue them for asynchronous execution via message brokers like RabbitMQ or Redis.
An async Redis client with full type hints, supporting cluster and sentinel deployments, built on anyio for compatibility with asyncio and trio.
Adds Redis-based message brokers and result backends to taskiq, enabling distributed task queuing and result storage with support for streams, pub/sub, and list-based queue patterns.
Install it if you need distributed task queuing with Redis and are already using or planning to adopt taskiq.
Provides async/await bindings for the Docker HTTP API, letting you manage containers, images, and other Docker resources from Python asyncio code.
Coiled is a Python client library that provisions and manages Dask clusters on cloud infrastructure, handling resource allocation, networking, and environment setup to scale parallel workloads.
However, verify the Elastic-2.0 license terms for your use case, and confirm whether coiled.io's service requires a paid account—the client library alone is not…
Hatchet SDK is the official Python client for Hatchet, a distributed task queue that lets you define, schedule, and orchestrate workflows with automatic retries and event-driven triggers.
Install it if you're already running a Hatchet server or planning to; otherwise, verify that Hatchet's architecture fits your deployment model.
An asynchronous Python framework for building event-driven applications with support for multiple message brokers and automatic AsyncAPI documentation generation.
Install it if you are building message-driven microservices and want to reduce boilerplate.
Pulumi Kubernetes Resource Provider lets you define, deploy, and manage Kubernetes API resources and workloads programmatically using Python, with full API coverage including Deployments, Services, ConfigMaps, and other Kubernetes primitives.
Install it if you are already using Pulumi and need to deploy or manage Kubernetes workloads programmatically; it is not a standalone Kubernetes tool and requires…
Python SDK for publishing and consuming messages on Vercel Queues, with built-in adapters for Celery, Dramatiq, and Django task systems.
Implements rendezvous hashing to locate objects across a cluster of machines, minimizing data movement when cluster topology changes.
However, verify compatibility with your Python version (support beyond 3.8 is unspecified) and be aware that no active maintenance means no future bug fixes or…
Publishes Python dictionaries as JSON messages to AWS SNS topics, automatically converting dict fields into SNS message attributes.
Python client library for HashiCorp Consul that provides programmatic access to Consul's key-value store, service catalog, health checks, and ACL management through a synchronous and asynchronous API.
Install it if you need to integrate Consul into a Python application for service discovery, configuration management, or health monitoring.
Docker Compose is a command-line tool for defining and running multi-container Docker applications using a declarative YAML configuration file.
However, the latest release date (2021-05-10) is significantly outdated.
Provides Python bindings to XRootD, a high-performance distributed filesystem client for accessing data across remote storage resources with fault tolerance and transparent protocol handling.
Ypy is a Python binding for Y-CRDT that provides distributed data types enabling real-time collaboration and state synchronization between devices through a transaction-based API.
Provides GCP-specific integrations for Dagster, enabling data pipelines to interact with Google Cloud services like BigQuery and Cloud Storage.
Django-RQ integrates RQ (Redis Queue) with Django, allowing you to configure and manage asynchronous job queues through Django settings and enqueue tasks for background processing.
Simpleflow provides a Python library for writing distributed dataflow programs that coordinate task execution through Amazon SWF, using a futures-based abstraction to define task dependencies.
Anyscale provides a command-line interface and Python SDK for managing workloads and services on the Anyscale platform, a distributed computing service.
However, verify the 'AS License' terms first, as the license treatment is unclear and may impose restrictions.
Metaflow is a framework for building, prototyping, and deploying AI and ML systems, handling the full lifecycle from local development through production orchestration with support for scaling across cloud compute resources.
mpi4py provides Python bindings for the Message Passing Interface (MPI) standard, enabling distributed and parallel computing across multiple processes and machines.
A Python client library for Kubernetes that mirrors kubectl's command patterns, offering both sync and async APIs to query, create, and manage cluster resources.
Registers a Spark data source that reads datasets from Hugging Face and writes Spark DataFrames back to Hugging Face as Parquet, with support for splits, configs, filtering, and deduplication.
Install it if you need to move datasets between Spark and Hugging Face, or if you want to use Hugging Face datasets in a Spark pipeline.
Provides tools for distributing Pandas DataFrames and Series across Apache Spark clusters for large-scale data processing.
wmill is a Python client library for the Windmill workflow automation platform, providing methods to access variables, resources, run scripts, and manage job state.
Install it if you need to interact with a Windmill instance from Python.
Provides REST API resource integrations for Dagster data pipelines, enabling HTTP-based asset definitions and orchestration within Dagster's declarative asset framework.
Provides a Python client library for YTsaurus, enabling programmatic access to distributed storage, MapReduce operations, and NoSQL key-value storage on a YTsaurus cluster.
Provides gRPC protocol buffer definitions for communicating with the Isolate Cloud controller service.
However, verify the license status before use in proprietary contexts, and confirm that your use case actually requires direct gRPC communication with Isolate Cloud…
Pypeln lets you build concurrent data pipelines using processes, threads, or asyncio tasks with a unified functional API, allowing fine-grained control over workers and memory at each stage.
However, avoid it for new projects requiring long-term support or compatibility with future Python versions—the abandoned status means no bug fixes or updates.
Stores checkpoints and partition ownership for Azure Event Hubs consumers using Azure Blob Storage as the persistent backend, enabling async event processing with automatic failover and replay capability.
Install it if you are building an async Event Hubs consumer application and need reliable checkpoint persistence—it is the standard way to do so on Azure.
Provides Python bindings for a high-performance distributed key-value cache engine designed for LLM and graph reasoning inference workloads, with multi-tier memory pooling and cross-device data access.
Displays multiple tqdm progress bars from worker processes in a multiprocessing pool, with logging and aggregate progress monitoring routed through the main process.
A high-performance Rust-based load balancer and gateway for routing requests across distributed SGLang model workers, supporting multiple load-balancing strategies, prefill-decode disaggregation, and OpenAI-compatible APIs.
Not recommended for simple single-worker or non-SGLang deployments.