Packages
Integrates Kafka with Confluent Schema Registry to manage topic creation, schema publishing, and message serialization in a single workflow.
However, be aware that the package has not been updated since mid-2022; if you depend on recent Kafka or Schema Registry features, or if you need active maintenance…
Provides multi-node multi-GPU distributed computing infrastructure for building GPU-accelerated algorithms with Dask, wrapping RAFT's host-accessible runtime APIs.
Not recommended for single-node work or exploratory data science—use pylibraft-cu12 directly or RAPIDS cuML instead.
A Python client library for interacting with Netflix Genie, a distributed job execution service, providing resource models, API wrappers, and retry logic for submitting and managing jobs.
However, maintenance is aging (last release 528 days ago), so verify that it remains compatible with your Genie instance version and modern Python environments before…
Mitogen is a library for writing distributed self-replicating programs that coordinate work across multiple machines or processes.
However, the minimal documentation excerpt makes it difficult to assess fit without consulting the full docs—verify that the library's specific capabilities match…
Extends the Dapr Python SDK with workflow authoring capabilities, enabling you to define and execute distributed workflows using Dapr's orchestration runtime.
oslo.versionedobjects provides a versioned object model for distributed systems that handles serialization, field typing, and RPC-friendly method calls to maintain compatibility across service upgrades.
Provides a Python interface to Globus APIs for managing data transfer, authentication, and access to Globus services.
Install it if you need to integrate Globus data transfer or authentication into a Python application.
Pyro4 enables Python objects to communicate over a network using normal method calls, handling remote object location and serialization automatically.
TaskFlow is a library for defining and executing workflows as structured state machines, with support for task composition, error handling, persistence, and distributed execution across multiple workers.
Install it if you're building task-driven systems; skip it if you only need simple job queuing or one-off script execution.
VMware vAPI Runtime provides HTTP client, message serialization, and input/output validation for invoking VCF SDK APIs from Python applications.
Flyte is a Python SDK for defining, orchestrating, and executing ML pipelines and services at scale, supporting both async and synchronous task definitions with built-in retries, caching, and local or remote execution.
However, the license treatment is unclear in the metadata despite Apache 2.0 being mentioned in the description—verify the actual license before production use.
Provides Python bindings to VMware vCenter Server APIs for managing virtual machines, content libraries, resource deployment, tagging, and security certificates.
Install it if you need to automate vCenter operations or integrate vSphere into Python tooling.
p_tqdm wraps pathos and tqdm to apply a function in parallel across multiple inputs while displaying a live progress bar with time estimates.
Provides Python client bindings for VMware vAPI common services, enabling programmatic interaction with VMware Cloud Foundation infrastructure components.
Python client library and CLI for the OpenStack Bare Metal API, providing programmatic and command-line interfaces to manage bare metal provisioning and node operations.
RepartiPy calculates optimal partition counts for PySpark DataFrames and estimates their size using execution plan statistics, supporting both memory-cached and disk-based sampling approaches.
However, the dormant maintenance status since 2024-03-08 means no bug fixes or compatibility updates are forthcoming—install only if you can tolerate a static…
Adds a progress bar to joblib.Parallel execution using tqdm, displaying real-time completion status as parallel jobs run.
However, the unclear license status (CC-BY-SA-4.0 is not a standard software license) and aging maintenance cycle (362 days since last release) warrant caution in…
Provides a high-performance UCX communication backend for Dask Distributed, enabling GPU-to-GPU communication via NVLink, InfiniBand, and other high-speed interconnects.
Provides Python client library for managing Azure Stream Analytics resources—jobs, inputs, outputs, functions, and clusters—via Azure's REST API.
Install it if you need to programmatically manage Stream Analytics resources on Azure.
Clusterscope extracts and queries information from HPC clusters and jobs via a CLI tool and Python library, with built-in support for Slurm cluster metadata and AWS/NCCL diagnostics.
Outerbounds is the main platform package that integrates data science workflows with cloud infrastructure, orchestrating machine learning jobs across Kubernetes and AWS environments.
However, the proprietary license requires careful review of terms before production use, and the 8 runtime dependencies add installation complexity.
pyhelm3 is a Python library for programmatically managing Helm releases in Kubernetes clusters using Helm 3 and later, supporting installation, upgrades, listing, and uninstallation of releases.
Python SDK for creating, managing, and executing workloads in Blaxel's perpetual sandbox platform, with support for process execution, filesystem operations, volumes, and preview URLs.
However, verify the license terms first—they are not declared in the package metadata—and ensure your use case aligns with Blaxel's pricing and authentication model.
torchrunx distributes PyTorch training functions across multiple GPUs or machines using a functional API, eliminating the need for CLI-based launchers.
Implements the RedLock algorithm to provide distributed locks across multiple independent Redis nodes, preventing single-point-of-failure issues that arise with single-master Redis lock implementations.
Serializes and deserializes PyTorch modules and tensors to/from HTTP, HTTPS, S3, Redis, and local filesystem endpoints with streaming support for fast model loading.
Automates IBM Quantum backend connection, authentication, and selection by detecting your subscription plan and returning the least-busy available QPU resource for immediate use in Qiskit circuits.
However, note the aging maintenance status (219 days since last release) and verify that the connector's error handling and backend-selection logic align with your…
Tooz provides a coordination API for distributed systems, centralizing common primitives like group membership, lock services, and leader election.
Install it if you're building distributed applications that need coordination primitives.
Mooncake Transfer Engine provides high-performance, zero-copy RDMA-based data transfer for distributed KV cache management in large-scale LLM inference and training systems.
Provides C++ bindings for serializing and deserializing YSON format data, enabling Python code to work with YTsaurus's native data format.
However, verify Python version compatibility first (not declared in metadata), and be aware the package is aging with no recent maintenance—suitable for stable,…
Provides Python bindings to Qpid Proton, an AMQP 1.0 messaging library for building messaging applications like brokers, clients, routers, and bridges.
Provides Airflow operators and hooks to trigger and monitor Hightouch data sync runs from Airflow workflows.
Install only if you have apache-airflow >= 1.10 and a Hightouch workspace with API access.
Integrates MLflow experiment tracking and model management with Dagster's data orchestration, allowing you to track ML model training and artifacts as part of your Dagster asset pipelines.
A Rust-compiled workflow orchestration engine that executes the same API as the open-source kailash library, optimized for performance and memory efficiency.
Bundles Apache Flink's required libraries for Python, enabling distributed stream and batch processing computations across cluster environments.
Streams time-series data to Nominal Core with optional file fallback and backpressure handling, wrapping a Rust streaming client for Python.
However, be aware that the library is under active development and may introduce breaking changes; review the warning in its documentation and test thoroughly before…
Snowpark Connect for Spark lets you run Spark workloads directly against Snowflake's compute engine using the Spark Connect protocol, without managing a separate Spark cluster.
Provides Kafka topic definitions and JSON schema validators for Sentry's event streaming infrastructure, with auto-generated Python type hints for schema-compliant message handling.
Thespian provides an Actor Model framework for building concurrent, distributed, and fault-tolerant applications in Python where independent actors communicate via message passing.
However, install friction is high and the project shows signs of aging—no release in 520 days despite Production/Stable status.
kcli is a command-line provisioning and management tool for virtual machines across multiple virtualization providers (libvirt, KubeVirt, oVirt, OpenStack, VMware vSphere, AWS, Azure, GCP, IBM Cloud, Hcloud) using YAML plan files and cloud images.
However, verify Python version compatibility and system-level libvirt dependencies before installing, and clarify the license treatment if you need legal certainty…