apache-airflow-providers-apache-livy

What it is and what it does

This is an Apache Airflow provider package that adds Livy integration, allowing you to submit and manage Spark jobs running on remote Apache Livy servers as tasks in your Airflow DAGs. It bridges Airflow's orchestration capabilities with Spark clusters managed through Livy, enabling you to incorporate distributed Spark workloads into larger data pipelines without running Spark directly on the Airflow scheduler.

The package depends on apache-airflow, apache-airflow-providers-http, apache-airflow-providers-common-compat, and aiohttp. It is actively maintained by the Apache Airflow project, supports Python 3.10 through 3.14, and requires Airflow 2.11.0 or later. Installation is straightforward via pip and integrates seamlessly into existing Airflow deployments.

Use it for:

• Submit Spark batch jobs from Airflow DAGs to a shared Livy cluster without embedding Spark in the scheduler.
• Orchestrate multi-stage data pipelines where some tasks run on Spark via Livy and others use standard Airflow operators.
• Monitor remote Spark job execution and integrate job status into Airflow's task dependency graph.
• Centralize Spark job submission through Airflow for teams managing multiple Livy-based clusters.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Integrates Apache Livy with Apache Airflow to submit and monitor Spark jobs on remote Livy servers as part of Airflow workflows.

Yes, if you run Apache Airflow and need to submit Spark jobs to remote Livy servers. The package is actively maintained, has no known vulnerabilities, installs with low friction, and is backed by the Apache Airflow project. Install it only if you have a Livy deployment in your infrastructure; it is not useful as a standalone tool.

Install

apache-airflow-providers-apache-livy on PyPI

pip install apache-airflow-providers-apache-livy

uv add apache-airflow-providers-apache-livy

poetry add apache-airflow-providers-apache-livy

Installing apache-airflow-providers-apache-livy

Before you install

Low friction install as a pure Python wheel. Actively maintained with a recent release (6 days old) and backed by the Apache Airflow project's infrastructure.

License in practice

Apache-2.0 permissive license allows use in commercial and proprietary projects with minimal restrictions.

Quickstart

pip install apache-airflow-providers-apache-livy

from airflow.providers.apache.livy.operators.livy import LivyOperator

livy_task = LivyOperator(
    task_id='spark_job',
    file='s3://path/to/job.jar',
    livy_conn_id='livy_default'
)

Requires Apache Airflow >=2.11.0 and a running Apache Livy server accessible from the Airflow environment.

Verify before relying

• Whether the package supports Livy session management (interactive vs. batch modes) beyond basic job submission.
• What monitoring and retry capabilities are available for long-running Spark jobs.
• Whether it handles Livy server authentication methods beyond basic HTTP connection configuration.

Package facts

Evidence: apache_airflow_providers_apache_livy-4.6.0-py3-none-any.whl

Keywords: airflow-provider, apache.livy, airflow, integration

Tags

More Monitoring packages