Packages
Tensorlake provides a Python SDK for building agentic applications using isolated Firecracker MicroVM sandboxes and a serverless orchestration runtime with automatic scaling.
However, verify the unclear license status before production use, and confirm pricing and scaling limits align with your workload.
Cosmos-xenna is a Python framework for building and running distributed AI inference pipelines on Ray clusters, handling resource allocation, autoscaling, and data flow across multi-stage workflows.
However, the package description explicitly states it is no longer under active development and recommends migration to Cosmos 3 for new projects.
SkyPilot provisions and manages AI compute jobs across multiple cloud providers, Kubernetes clusters, Slurm systems, and on-premises infrastructure through a unified YAML or Python interface.
Deploys Dask distributed computing clusters on job queuing systems like PBS, Slurm, SGE, and LSF, automating the submission and management of worker nodes.
Rucio-clients provides command-line and programmatic access to Rucio, a distributed scientific data management framework that organizes, replicates, and retrieves large volumes of data across globally distributed storage systems.
Provides AI assistants with programmatic access to Apache Airflow via the Model Context Protocol, exposing DAG management, task execution, monitoring, and configuration tools through a FastMCP server.
OSSFS provides a Python interface to Alibaba Cloud's Object Storage Service (OSS) through the fsspec standard API, enabling file-like operations on OSS objects with both synchronous and asynchronous backends.
Provides tools and APIs for publishing and consuming messages on AMQP brokers, with schema declaration and async consumer services for Fedora's messaging infrastructure.
However, be aware of the GPLv2+ copyleft license (verify dual-licensing implications for proprietary code) and the requirement for a running AMQP broker.
A message relay service that subscribes to SQS queues and delivers notifications via email (SES), Slack, DataDog, and other endpoints, designed to work with Cloud Custodian policies.
Python bindings and command-line tool for the OpenStack Magnum API, enabling programmatic management of container clusters.
Install only if you have a Magnum deployment or plan to use OpenStack container services.
Provides client-side extensions and integrations for Metaflow workflows on the Outerbounds platform, enabling cloud orchestration and execution capabilities.
However, verify that the Commercial license terms align with your use case and confirm that your organization has or intends to use an Outerbounds platform subscription.
Python client library for Consul, providing programmatic access to Consul's HTTP API for service discovery, key-value storage, and health checking.
Pulsar Galaxy Lib is a Python server that enables Galaxy to run jobs on remote systems without requiring shared file systems, handling file transfer, job execution, and result retrieval automatically.
Provides a standardized collection of resource class symbols for OpenStack deployments, representing distinct inventory types like VCPU and DISK_GB.
However, be aware it is abandoned—do not expect bug fixes or updates if OpenStack's resource class definitions evolve.
RayDP runs Apache Spark on Ray and integrates Spark with AI libraries, enabling you to build distributed data and machine learning pipelines in a single Python program.
Manages discovery and removal of storage volumes attached to a host, supporting multiple transport protocols for OpenStack Cinder integration.
CodeFlare SDK provides a Python interface for submitting and managing batch jobs and resource requests on Kubernetes clusters, supporting multiple authentication methods and simplifying access to high-performance compute resources.
Provides Netflix-developed extensions to Metaflow, including an improved Conda environment decorator that mixes Conda and PyPI packages, debugging tools, and Hugging Face authentication support.
However, be aware that this is experimental functionality without the stability guarantees of core Metaflow; features may change or be removed if merged upstream.
Provides Pydantic models and a lightweight HTTP client for the GA4GH Task Execution Service (TES) specification, enabling structured interaction with TES-compliant task execution systems.
Salt is an event-driven automation engine for deploying, configuring, and managing infrastructure at scale—servers, virtual machines, containers, network devices, and more—through remote execution and configuration management.
Vineyard is an in-memory immutable data manager that enables zero-copy sharing of distributed data across big data systems like graph analytics, numerical computing, and machine learning frameworks.
However, install only if you have a Vineyard server infrastructure in place and are working at scale; it is not a general-purpose data library for single-machine use.
Provides version checking and validation utilities for distributed data platform components, plus data interface abstractions for charm libraries.
Python client library and command-line interface for the Mistral workflow service REST API, enabling programmatic and CLI-based interaction with Mistral workflow management.
Install only if you have a Mistral server running and need to interact with it programmatically or via CLI; it is not useful standalone.
Provides per-step fault tolerance for PyTorch distributed training, allowing jobs to recover from worker failures without restarting the entire training process.
However, verify the BSD 3-Clause license applies to your use case (metadata is unclear), and be aware that medium install friction (Rust compilation, system…
Vineyard is an in-memory immutable data manager that enables zero-copy sharing of distributed data across big data systems like graph analytics, numerical computing, and machine learning frameworks.
Provides a Python interface to Open MPI, a high-performance implementation of the Message Passing Interface (MPI) specification for distributed computing and parallel applications.
The main gotcha is that it requires the Open MPI system library as a prerequisite and does not specify Python version support, so verify compatibility with your…
Official Python SDK for Render that lets you define and run workflow tasks, manage task runs, and access experimental features like object storage and key-value services.
Integrates Fivetran data connectors with Dagster's orchestration engine, enabling declarative asset definitions that automatically sync data from Fivetran sources.
Provides compiled EPICS Core libraries and development headers for Python, enabling Python modules to interface with EPICS control systems without requiring separate system-level EPICS installation.
Parallelbar wraps Python's multiprocessing Pool methods (map, starmap, imap, imap_unordered) to display live progress bars and handle exceptions and timeouts across worker processes.
Metaflow extension that pre-bakes conda and PyPI dependencies into Docker images at deploy time, eliminating conda bootstrap overhead when tasks start on remote runners.
Install only if you have a registry and build service already available or planned; it adds no value for local-only flows.
Migrates Airflow metadata—Variables, Connections, Environment Variables, Pools, and DAG History—between two Airflow instances.
Provides decorators for Metaflow workflows to save and resume checkpoints, manage models, and load external models from Hugging Face Hub, enabling long-running jobs to recover from failures without losing progress.
Provides Python bindings to the OpenNebula XML-RPC API, enabling programmatic control of OpenNebula cloud infrastructure through Python code.
Install only if you have an OpenNebula server to connect to; it is not useful as a standalone library.
Wraps Apache Spark Connect to let Python applications connect to remote Dataproc Spark sessions via the Spark Connect protocol without extra setup steps.
oslo.limit enforces quota and rate limits across OpenStack services by calculating and validating resource consumption against configured quotas.
oslo.vmware provides a library for common VMware operations and APIs, enabling OpenStack components to interact with VMware infrastructure.
Integrates Hydra configuration framework with Submitit to submit Python functions as Slurm cluster jobs directly from configuration-driven applications.
However, note that the package has not been updated since May 2022; verify compatibility with your versions of hydra-core and submitit before committing to production…
Provides Python bindings and utilities for Apache Spark, including dataset diffing, histograms, global row numbering, Parquet inspection, and dynamic Python package installation into running PySpark jobs.
Python SDK to create, manage, and automate Lightning AI resources—compute, studios, jobs, deployments, and sandboxes—from scripts and notebooks without leaving your code.