---
id: apache-airflow-providers-apache-livy
version: "4.6.0"
license: Apache-2.0
license_treatment: permissive
maintenance: active
---

# apache-airflow-providers-apache-livy — Provider package apache-airflow-providers-apache-livy for Apache Airflow

License: permissive · Maintenance: active · Downloads: 641.6K/mo

## What it is and what it does

This is an Apache Airflow provider package that adds Livy integration, allowing you to submit and manage Spark jobs running on remote Apache Livy servers as tasks in your Airflow DAGs. It bridges Airflow's orchestration capabilities with Spark clusters managed through Livy, enabling you to incorporate distributed Spark workloads into larger data pipelines without running Spark directly on the Airflow scheduler.

The package depends on apache-airflow, apache-airflow-providers-http, apache-airflow-providers-common-compat, and aiohttp. It is actively maintained by the Apache Airflow project, supports Python 3.10 through 3.14, and requires Airflow 2.11.0 or later. Installation is straightforward via pip and integrates seamlessly into existing Airflow deployments.

Use it for:

- Submit Spark batch jobs from Airflow DAGs to a shared Livy cluster without embedding Spark in the scheduler.
- Orchestrate multi-stage data pipelines where some tasks run on Spark via Livy and others use standard Airflow operators.
- Monitor remote Spark job execution and integrate job status into Airflow's task dependency graph.
- Centralize Spark job submission through Airflow for teams managing multiple Livy-based clusters.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Integrates Apache Livy with Apache Airflow to submit and monitor Spark jobs on remote Livy servers as part of Airflow workflows.

Yes, if you run Apache Airflow and need to submit Spark jobs to remote Livy servers. The package is actively maintained, has no known vulnerabilities, installs with low friction, and is backed by the Apache Airflow project. Install it only if you have a Livy deployment in your infrastructure; it is not useful as a standalone tool.

## Install

pip install apache-airflow-providers-apache-livy
uv add apache-airflow-providers-apache-livy
poetry add apache-airflow-providers-apache-livy

## Installing apache-airflow-providers-apache-livy

Before you install: Low friction install as a pure Python wheel. Actively maintained with a recent release (6 days old) and backed by the Apache Airflow project's infrastructure.

License in practice: Apache-2.0 permissive license allows use in commercial and proprietary projects with minimal restrictions.

Quickstart:

pip install apache-airflow-providers-apache-livy

from airflow.providers.apache.livy.operators.livy import LivyOperator

livy_task = LivyOperator(
    task_id='spark_job',
    file='s3://path/to/job.jar',
    livy_conn_id='livy_default'
)

Requires Apache Airflow >=2.11.0 and a running Apache Livy server accessible from the Airflow environment.

Verify before relying:

- Whether the package supports Livy session management (interactive vs. batch modes) beyond basic job submission.
- What monitoring and retry capabilities are available for long-running Spark jobs.
- Whether it handles Livy server authentication methods beyond basic HTTP connection configuration.

## Package facts

- License: Apache-2.0 (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 641.6K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

airflow apache livy provider, spark job submission airflow, livy integration airflow, remote spark execution airflow, airflow livy operator, airflow-provider, spark-integration, orchestration

[View on SkillFed](https://skillfed.io/packages/apache-airflow-providers-apache-livy) · [View on PyPI](https://pypi.org/project/apache-airflow-providers-apache-livy/)