{"categories":[{"label":"Monitoring","url":"https://skillfed.io/packages/category/system-monitoring/3"}],"enrichment":{"capability":"Integrates Apache Livy with Apache Airflow to submit and monitor Spark jobs on remote Livy servers as part of Airflow workflows.","skillfed_tags":["airflow-provider","spark-integration","orchestration"],"use_cases":["Submit Spark batch jobs from Airflow DAGs to a shared Livy cluster without embedding Spark in the scheduler.","Orchestrate multi-stage data pipelines where some tasks run on Spark via Livy and others use standard Airflow operators.","Monitor remote Spark job execution and integrate job status into Airflow's task dependency graph.","Centralize Spark job submission through Airflow for teams managing multiple Livy-based clusters."],"what_it_does":"This is an Apache Airflow provider package that adds Livy integration, allowing you to submit and manage Spark jobs running on remote Apache Livy servers as tasks in your Airflow DAGs. It bridges Airflow's orchestration capabilities with Spark clusters managed through Livy, enabling you to incorporate distributed Spark workloads into larger data pipelines without running Spark directly on the Airflow scheduler.\n\nThe package depends on apache-airflow, apache-airflow-providers-http, apache-airflow-providers-common-compat, and aiohttp. It is actively maintained by the Apache Airflow project, supports Python 3.10 through 3.14, and requires Airflow 2.11.0 or later. Installation is straightforward via pip and integrates seamlessly into existing Airflow deployments.","worth_installing":"Yes, if you run Apache Airflow and need to submit Spark jobs to remote Livy servers. The package is actively maintained, has no known vulnerabilities, installs with low friction, and is backed by the Apache Airflow project. Install it only if you have a Livy deployment in your infrastructure; it is not useful as a standalone tool."},"id":"apache-airflow-providers-apache-livy","links":{"html":"https://skillfed.io/packages/apache-airflow-providers-apache-livy","md":"https://skillfed.io/packages/apache-airflow-providers-apache-livy.md","pypi":"https://pypi.org/project/apache-airflow-providers-apache-livy/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-08-08","license_spdx":"Apache-2.0","license_treatment":"permissive","name":"apache-airflow-providers-apache-livy","python_support":"supports_current","summary":"Provider package apache-airflow-providers-apache-livy for Apache Airflow"},"popularity":{"monthly_downloads":641611,"position":5612,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"4.6.0"}