apache-airflow-providers-apache-livy
Provider package apache-airflow-providers-apache-livy for Apache Airflow
Decision gist · record as of 2026-08-14
Yes, if you run Apache Airflow and need to submit Spark jobs to remote Livy servers. The package is actively maintained, has no known vulnerabilities, installs with low friction, and is backed by the Apache Airflow project. Install it only if you have a Livy deployment in your infrastructure; it is not useful as a standalone tool.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Apache Airflow >=2.11.0 and a running Apache Livy server accessible from the Airflow environment.
- Low friction install as a pure Python wheel.
- Actively maintained with a recent release (6 days old) and backed by the Apache Airflow project's infrastructure.
License · maintenance · safety
Apache-2.0 (permissive) — Apache-2.0 permissive license allows use in commercial and proprietary projects with minimal restrictions.
last release 2026-08-08 (6 days) · last repo commit 2026-08-14 · 46,490 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 641,611 downloads/mo, #5,612 on PyPI
Alternatives
Verify before relying
pip install apache-airflow-providers-apache-livy
from airflow.providers.apache.livy.operators.livy import LivyOperator
livy_task = LivyOperator(
task_id='spark_job',
file='s3://path/to/job.jar',
livy_conn_id='livy_default'
)- Whether the package supports Livy session management (interactive vs. batch modes) beyond basic job submission.
- What monitoring and retry capabilities are available for long-running Spark jobs.
- Whether it handles Livy server authentication methods beyond basic HTTP connection configuration.
What it is and what it does
This is an Apache Airflow provider package that adds Livy integration, allowing you to submit and manage Spark jobs running on remote Apache Livy servers as tasks in your Airflow DAGs. It bridges Airflow's orchestration capabilities with Spark clusters managed through Livy, enabling you to incorporate distributed Spark workloads into larger data pipelines without running Spark directly on the Airflow scheduler.
The package depends on apache-airflow, apache-airflow-providers-http, apache-airflow-providers-common-compat, and aiohttp. It is actively maintained by the Apache Airflow project, supports Python 3.10 through 3.14, and requires Airflow 2.11.0 or later. Installation is straightforward via pip and integrates seamlessly into existing Airflow deployments.
Use it for
- Submit Spark batch jobs from Airflow DAGs to a shared Livy cluster without embedding Spark in the scheduler.
- Orchestrate multi-stage data pipelines where some tasks run on Spark via Livy and others use standard Airflow operators.
- Monitor remote Spark job execution and integrate job status into Airflow's task dependency graph.
- Centralize Spark job submission through Airflow for teams managing multiple Livy-based clusters.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you run Apache Airflow and need to submit Spark jobs to remote Livy servers.
The package is actively maintained, has no known vulnerabilities, installs with low friction, and is backed by the Apache Airflow project. Install it only if you have a Livy deployment in your infrastructure; it is not useful as a standalone tool.
Install
apache-airflow-providers-apache-livy on PyPI
Before you install
Low friction install as a pure Python wheel. Actively maintained with a recent release (6 days old) and backed by the Apache Airflow project's infrastructure.
Requires Apache Airflow >=2.11.0 and a running Apache Livy server accessible from the Airflow environment.
License in practice
Apache-2.0 permissive license allows use in commercial and proprietary projects with minimal restrictions.
Quickstart
pip install apache-airflow-providers-apache-livy
from airflow.providers.apache.livy.operators.livy import LivyOperator
livy_task = LivyOperator(
task_id='spark_job',
file='s3://path/to/job.jar',
livy_conn_id='livy_default'
)Verify before relying
- Whether the package supports Livy session management (interactive vs. batch modes) beyond basic job submission.
- What monitoring and retry capabilities are available for long-running Spark jobs.
- Whether it handles Livy server authentication methods beyond basic HTTP connection configuration.
Package facts
| License | Apache-2.0 permissive |
| Python support | Supports the current Python release >=3.10 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 4 packagesapache-airflowapache-airflow-providers-httpapache-airflow-providers-common-compataiohttp |
| Maintenance | Actively maintained 6 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 641,611 / month, #5,612 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 5 - Production/StableEnvironment :: ConsoleEnvironment :: Web EnvironmentFramework :: Apache AirflowFramework :: Apache Airflow :: ProviderIntended Audience :: DevelopersIntended Audience :: System AdministratorsProgramming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Topic :: System :: Monitoring |
Evidence: apache_airflow_providers_apache_livy-4.6.0-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “airflow apache livy provider”
- apache-airflow-providers-apache-livyIntegrates Apache Livy with Apache Airflow to submit and monitor…
- apache-airflow-providers-apache-drillProvides Apache Airflow integration for Apache Drill, enabling…
- apache-airflow-providers-apache-flinkProvides Apache Airflow integration for Apache Flink, enabling you to…
Give your agent the search over MCP, or paste the wish link into any chat.
More Monitoring packages
Wraps any iterable to display a real-time progress bar in the terminal or Jupyter notebook, showing iteration count, elapsed time, and estimated time remaining.
Provides generated Python code for OpenTelemetry semantic conventions, enabling standardized attribute naming and constant definitions for instrumentation and telemetry collection.
Install it if you are using OpenTelemetry and want to follow semantic conventions correctly.
Provides the reference implementation of the OpenTelemetry API for collecting and exporting traces, metrics, and logs from Python applications.
Provides the abstract API and interfaces for OpenTelemetry instrumentation in Python, defining how to emit traces, metrics, and logs without tying code to a specific SDK implementation.
Exports OpenTelemetry observability data to an OpenTelemetry Collector using Protobuf-encoded messages over HTTP.
Install it if you are using OpenTelemetry in Python and need to send data to a Collector over HTTP.
Provides automatic instrumentation commands and programmatic APIs to inject distributed tracing into Python applications without code changes, detecting and instrumenting packages used by your program.
Install it if you need distributed tracing without code changes and have compatible instrumented packages in your environment.
See also apache-airflow-providers-informatica · livy · apache-airflow-providers-asana · apache-airflow-providers-jenkins · apache-airflow-providers-atlassian-jira · apache-airflow-providers-airbyte · apache-airflow-providers-git · apache-airflow-providers-ssh · apache-airflow-providers-apache-spark · apache-airflow-providers-segment