Packages
gRPC Python is an HTTP/2-based RPC framework that enables you to define and call remote procedures across network boundaries using protocol buffers for serialization.
Install it if you need RPC communication in a distributed system or are integrating with existing gRPC services.
execnet lets you spawn and communicate with Python interpreters across local processes, remote hosts, and different platforms, using a simple API for task distribution and inter-process messaging.
However, the aging maintenance status (275 days since last release) means you should verify it meets your concurrency and performance needs before committing to a…
Cloudpickle extends Python's standard pickle module to serialize lambda functions, interactively-defined functions and classes, and other constructs that the default pickle cannot handle, making it suitable for cluster computing and remote code execution.
Install it if you need to serialize lambda functions, interactively-defined code, or non-standard Python constructs for cluster computing or distributed execution.
Provides a unified, open()-compatible Python API for streaming large files from remote storage (S3, GCS, Azure, HDFS, SFTP, HTTP) and local filesystems, with transparent compression support.
Install it if you work with large files on cloud storage or remote systems and want to avoid writing boilerplate around multiple SDKs.
Portalocker provides cross-platform file locking with support for exclusive and shared locks, plus Redis-based distributed locks and process-aware PID file locking.
Install it if you need file or process coordination; the optional extras (pywin32, redis) are only required for specific lock types.
Ray is a distributed computing framework that scales Python applications from a single machine to multi-node clusters, providing abstractions for parallel tasks, stateful actors, and shared objects.
Kombu is a messaging library that provides a high-level Python interface to AMQP and other message brokers, supporting pluggable transports for RabbitMQ, Redis, MongoDB, Amazon SQS, and others.
Install it if you are building any distributed system that requires reliable message passing.
Modal is a Python client library that lets you run Python functions on serverless cloud compute infrastructure directly from your local scripts, handling deployment, scaling, and execution on Modal's platform.
Celery is a distributed task queue that lets you run Python functions asynchronously across multiple workers, communicating through message brokers like RabbitMQ or Redis.
Install it if you need background job processing, task scheduling, or distributed work coordination.
billiard is a fork of Python's multiprocessing module with bug fixes and improvements, providing enhanced process pool management and inter-process communication for parallel task execution.
PySpark provides Python bindings to Apache Spark, enabling distributed data processing and analytics across large datasets using Spark's unified engine with support for SQL, machine learning, graph processing, and stream processing.
Temporal Python SDK provides a framework for building durable, distributed workflows and activities that execute asynchronously and survive failures through automatic retry and state replay.
Dask is a parallel computing library that scales Python analytics workloads across multiple cores or machines using task scheduling and lazy evaluation.
Provides a Python client library to submit, manage, and monitor large-scale parallel and high-performance computing batch jobs on Azure Batch.
Install it if you need to run distributed batch workloads on Azure; the main consideration is that v15.x is a significant departure from v14.x, so review the…
Nexus Python SDK provides a framework for defining synchronous RPC services and handlers that support both inline and asynchronous responses with operation tracking and cancellation.
However, be aware this is experimental software—expect breaking changes before a stable release, and verify that the SDK's current feature set fits your deployment…
Azure Service Bus client library for Python that enables sending and receiving messages through cloud-managed queues, topics, and subscriptions with support for asynchronous messaging patterns.
Install it if you need to integrate with Azure's messaging service or build distributed, asynchronous messaging patterns on Azure.
Provides Python bindings to manage Azure compute resources—virtual machines, scale sets, disks, galleries, and related infrastructure—through the Azure Resource Manager API.
Install it if you need to manage Azure compute resources programmatically; it is the standard choice for that task.
Docket is a distributed background task system that queues Python async functions for immediate or scheduled execution via Redis streams, with a unified interface for both immediate and future work.
Extends gRPC with connection pooling and affinity-based routing for Google Cloud Platform services, managing multiple channels and binding requests to specific backend instances.
Install only if maintaining legacy code that already depends on it.
Provides a filesystem interface to Azure Blob Storage and Azure Data Lake Storage Gen2, enabling file-like access through fsspec.
Locust is a Python-based load testing framework that lets you write performance tests in regular Python code and run them against HTTP and other protocols, with real-time monitoring via a web UI or command line.
Install it if you need load testing with code-based test definition, distributed scaling, and real-time monitoring—especially for HTTP services or when you want to…
Represents and manipulates IPv4, IPv6, MAC addresses and related network objects; supports CIDR notation, subnetting, set operations, IANA lookups, and DNS reverse generation.
Publishes and consumes events from Azure Event Hubs, a scalable pub-sub service for ingesting and streaming millions of events per second to multiple consumers.
Install it if you need to publish or consume events from Azure Event Hubs; the only caveat is that thread and coroutine safety are not guaranteed, so your application…
Flower is a web-based monitoring and management dashboard for Celery task clusters, providing real-time visibility into worker status, task execution, and remote control capabilities.
Install it if you run Celery in production or development and need visibility into task execution.
Ansible is an IT automation system that handles configuration management, application deployment, cloud provisioning, ad-hoc task execution, network automation, and multi-node orchestration using agentless SSH-based communication.
Install it if you need agentless configuration management or orchestration.
RQ is a Python library for queueing jobs and processing them in the background with workers, backed by Redis or Valkey. It handles job scheduling, prioritization, retries, and webhooks with a simple API.
Databricks Connect is a client library that lets you write Spark code locally in your IDE or notebook and execute it remotely on a Databricks cluster instead of running it locally.
Connects Dagster data pipelines to PostgreSQL for asset storage, state management, and run tracking.
Install only if you have a PostgreSQL instance available and require multi-user or persistent storage; Dagster's default storage is sufficient for local development.
Dask Expressions provides query optimization for Dask DataFrames by encoding operations as an expression tree that is optimized before execution, replacing the earlier Dask DataFrame implementation.
Install it if you are using Dask DataFrames; it is the recommended path forward.
Provides Python bindings to manage Azure Container Service resources (AKS clusters, agent pools, and related infrastructure) via the Azure Resource Manager API.
Install it if you need to manage AKS clusters programmatically.
aiokafka is an asyncio-based client library for Apache Kafka that provides asynchronous producer and consumer APIs for publishing and consuming messages.
Client library for interacting with Apache Spark clusters in Azure Synapse Analytics, providing programmatic access to Spark job submission and monitoring.
However, verify that its API surface and feature set meet your needs before committing to it for new projects, given the gap since the last release.
Manages Azure role-based access control, role assignments, and authorization policies through the Azure Management API.
Install it if you need to automate or query Azure access control from Python.
Dagster-pipes provides a toolkit for running Dagster integrations and transform logic outside of the main Dagster process, enabling external execution of data pipelines.
Distributed provides a scheduler and runtime for parallel and distributed computation using Dask, enabling you to scale Python workloads across multiple machines or cores.
Provides a web UI for Dagster, a data pipeline orchestrator that manages asset definitions, execution, and observability through a browser-based interface.
Provides a GraphQL API interface for querying and interacting with Dagster data pipelines, assets, and orchestration metadata.
Manages Azure Service Bus resources (namespaces, queues, topics, subscriptions, rules) programmatically via the Azure Resource Manager API.
Install it if you need to manage Service Bus resources programmatically on Azure—it is the standard tool for that task.
Stores Celery periodic task schedules in a Django database and manages them through the Django Admin interface, replacing static configuration with dynamic, editable task scheduling.
Install it if you use Celery with Django and need dynamic task scheduling; the low install friction and permissive license make adoption straightforward.
Manages Azure Event Hub resources—namespaces, hubs, consumer groups, and authorization rules—via the Azure management API.