Packages
Provides ETL/ELT integration for Dagster, enabling data pipeline orchestration with embedded extraction and loading capabilities through dagster-dlt and dagster-sling connectors.
Provides a high-performance Message Passing Interface (MPI) implementation for parallel computing applications, with support for multiple network modules and process management.
Total Perspective Vortex (TPV) provides dynamic job routing rules for Galaxy that direct tools, users, and roles to appropriate compute destinations based on configurable YAML policies.
Install it if Galaxy's static job configuration is limiting your deployment; skip it if you run a minimal single-machine Galaxy.
Provides SSH resource and I/O manager components for Dagster data pipelines, enabling remote execution and data transfer over SSH connections.
Provides a Python client to connect to OpenVoiceOS messagebus, send messages, and register handlers to react to messages from the OVOS system.
Install it if you need to communicate with an OVOS instance from Python or build OVOS integrations.
Provides a command-line interface to the OpenStack Placement service through the OpenStackClient plugin system, enabling CLI-based interaction with resource placement operations.
Install only if you are already using OpenStackClient and need placement-specific commands; it is not useful outside an OpenStack context.
An Azure plugin for Cloud Custodian that automates policy-driven governance and compliance across Azure resources through declarative YAML policies.
Install it if you need policy-driven compliance, cost control, or resource lifecycle automation.
Reports hyperparameter tuning metrics to Google CloudML Engine's hyperparameter tuning service, writing metric data to a local file for the service to consume.
Embeds a Rust-based federation node that composes cryptographic attestation, replication, and verification into a Python-callable service for decentralized mesh networks.
However, scalability claims are modeled rather than proven, and the package's maturity is unproven—treat as experimental for production use.
Integrates Sling ETL/ELT tasks into Dagster data pipelines, enabling declarative data asset orchestration with Sling as the execution engine.
Provides a Metaflow decorator that integrates PyTorch distributed training via torchrun, allowing you to run multi-node torch programs as parallel Metaflow tasks without modifying your training code.
Spark is an async-first actor framework for Python that lets you build concurrent and distributed systems by creating actors, sending messages with `tell()` (fire-and-forget) or `ask()` (request-reply), and managing them through a `Syndicate` instance.
Dask-GeoPandas parallelizes geospatial operations by combining GeoPandas' spatial capabilities with Dask's distributed computing, allowing you to process large geographic datasets across multiple partitions.
Install it if you work with large geographic datasets and need parallel processing.
Provides custom Apache Spark data sources using the Python Data Source API (Spark 4.0+), enabling you to read from and write to external systems directly as Spark DataFrames.
However, the aging maintenance status (200 days since last release) means you should verify that the specific data sources you need are actively maintained and…
pylxd provides a Python interface to the LXD REST API, enabling programmatic control of LXD containers and virtual machines.
Install it if you need to automate LXD operations from Python; skip it if you only use LXD from the command line or don't manage LXD systems.
Provides Apache Beam transforms for reading and writing relational databases, consuming and producing Kafka messages, and parsing CSV files and JSON data.
Routes and load-balances requests across vLLM worker instances with support for multiple algorithms, prefill-decode disaggregation, and Kubernetes service discovery.