---
id: apache-airflow-providers-apache-spark
version: "6.3.1"
license: Apache-2.0
license_treatment: permissive
maintenance: active
---

# apache-airflow-providers-apache-spark — Provider package apache-airflow-providers-apache-spark for Apache Airflow

License: permissive · Maintenance: active · Downloads: 1.2M/mo

## What it is and what it does

This is an Apache Airflow provider package that adds Spark integration to Airflow's orchestration framework. It allows you to define, schedule, and monitor Apache Spark jobs as tasks within Airflow DAGs, treating Spark workloads as first-class Airflow operators. The package depends on apache-airflow (>=2.11.0), pyspark-client (>=4.0.0), and several supporting libraries (grpcio-status, requests, tenacity) to handle communication, retries, and status reporting.

The provider is actively maintained by the Apache Airflow project, with production-stable status and support for Python 3.10 through 3.14. It is designed for developers and system administrators who need to integrate Spark batch or streaming jobs into larger Airflow-based data pipelines. Optional dependencies allow integration with Kubernetes clusters and OpenLineage data lineage tracking when needed.

Use it for:

- Schedule and monitor Spark batch jobs as part of a multi-step Airflow data pipeline.
- Orchestrate Spark SQL transformations triggered by upstream Airflow tasks.
- Submit Spark jobs to a Kubernetes cluster via Airflow using the cncf.kubernetes extra.
- Track data lineage of Spark jobs within Airflow using the openlineage optional dependency.
- Coordinate Spark workloads with non-Spark tasks in a unified Airflow workflow.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Integrates Apache Spark with Apache Airflow, enabling you to orchestrate and monitor Spark jobs as part of Airflow workflows.

Yes. This is a production-stable, actively maintained provider with low install friction and no known vulnerabilities. Install it if you run Airflow and need to orchestrate Spark jobs as part of your DAGs. Requires Airflow >=2.11.0 and Python >=3.10; verify your environment meets these minimums before installing.

## Install

pip install apache-airflow-providers-apache-spark
uv add apache-airflow-providers-apache-spark
poetry add apache-airflow-providers-apache-spark

## Installing apache-airflow-providers-apache-spark

Before you install: Low install friction; pure Python wheel. Active maintenance with release 6 days old. Requires apache-airflow >=2.11.0 and pyspark-client >=4.0.0 as core runtime dependencies.

License in practice: Apache-2.0 (permissive): you can use, modify, and distribute this package freely in commercial and private projects, provided you include a copy of the license and note any changes.

Quickstart:

pip install apache-airflow-providers-apache-spark

from airflow.providers.apache.spark import operators
# Use SparkSubmitOperator or other Spark operators in your DAG

Requires apache-airflow >=2.11.0 and Python >=3.10; pyspark-client >=4.0.0 must be installed.

Verify before relying:

- Whether pyspark-client >=4.0.0 is a drop-in replacement for traditional PySpark or requires different code patterns.
- Whether the optional extras (cncf.kubernetes, openlineage, pyspark) are commonly needed or edge cases.
- Specific Spark job types or configurations this provider supports or does not support.

## Package facts

- License: Apache-2.0 (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 1.2M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

airflow spark integration, orchestrate spark jobs airflow, spark provider airflow, airflow spark operator, schedule spark tasks airflow, spark dag airflow, airflow-provider, spark-integration, workflow-orchestration

[View on SkillFed](https://skillfed.io/packages/apache-airflow-providers-apache-spark) · [View on PyPI](https://pypi.org/project/apache-airflow-providers-apache-spark/)