apache-airflow-providers-apache-spark
Provider package apache-airflow-providers-apache-spark for Apache Airflow
Decision gist · record as of 2026-08-14
Yes. This is a production-stable, actively maintained provider with low install friction and no known vulnerabilities. Install it if you run Airflow and need to orchestrate Spark jobs as part of your DAGs. Requires Airflow >=2.11.0 and Python >=3.10; verify your environment meets these minimums before installing.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires apache-airflow >=2.11.0 and Python >=3.10; pyspark-client >=4.0.0 must be installed.
- Low install friction; pure Python wheel.
- Active maintenance with release 6 days old.
License · maintenance · safety
Apache-2.0 (permissive) — Apache-2.0 (permissive): you can use, modify, and distribute this package freely in commercial and private projects, provided you include a copy of the license and note any changes.
last release 2026-08-08 (6 days) · last repo commit 2026-08-14 · 46,490 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 1,187,469 downloads/mo, #4,246 on PyPI
Alternatives
Verify before relying
pip install apache-airflow-providers-apache-spark
from airflow.providers.apache.spark import operators
# Use SparkSubmitOperator or other Spark operators in your DAG- Whether pyspark-client >=4.0.0 is a drop-in replacement for traditional PySpark or requires different code patterns.
- Whether the optional extras (cncf.kubernetes, openlineage, pyspark) are commonly needed or edge cases.
- Specific Spark job types or configurations this provider supports or does not support.
What it is and what it does
This is an Apache Airflow provider package that adds Spark integration to Airflow's orchestration framework. It allows you to define, schedule, and monitor Apache Spark jobs as tasks within Airflow DAGs, treating Spark workloads as first-class Airflow operators. The package depends on apache-airflow (>=2.11.0), pyspark-client (>=4.0.0), and several supporting libraries (grpcio-status, requests, tenacity) to handle communication, retries, and status reporting.
The provider is actively maintained by the Apache Airflow project, with production-stable status and support for Python 3.10 through 3.14. It is designed for developers and system administrators who need to integrate Spark batch or streaming jobs into larger Airflow-based data pipelines. Optional dependencies allow integration with Kubernetes clusters and OpenLineage data lineage tracking when needed.
Use it for
- Schedule and monitor Spark batch jobs as part of a multi-step Airflow data pipeline.
- Orchestrate Spark SQL transformations triggered by upstream Airflow tasks.
- Submit Spark jobs to a Kubernetes cluster via Airflow using the cncf.kubernetes extra.
- Track data lineage of Spark jobs within Airflow using the openlineage optional dependency.
- Coordinate Spark workloads with non-Spark tasks in a unified Airflow workflow.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
This is a production-stable, actively maintained provider with low install friction and no known vulnerabilities. Install it if you run Airflow and need to orchestrate Spark jobs as part of your DAGs. Requires Airflow >=2.11.0 and Python >=3.10; verify your environment meets these minimums before installing.
Install
apache-airflow-providers-apache-spark on PyPI
Before you install
Low install friction; pure Python wheel. Active maintenance with release 6 days old. Requires apache-airflow >=2.11.0 and pyspark-client >=4.0.0 as core runtime dependencies.
Requires apache-airflow >=2.11.0 and Python >=3.10; pyspark-client >=4.0.0 must be installed.
License in practice
Apache-2.0 (permissive): you can use, modify, and distribute this package freely in commercial and private projects, provided you include a copy of the license and note any changes.
Quickstart
pip install apache-airflow-providers-apache-spark
from airflow.providers.apache.spark import operators
# Use SparkSubmitOperator or other Spark operators in your DAGVerify before relying
- Whether pyspark-client >=4.0.0 is a drop-in replacement for traditional PySpark or requires different code patterns.
- Whether the optional extras (cncf.kubernetes, openlineage, pyspark) are commonly needed or edge cases.
- Specific Spark job types or configurations this provider supports or does not support.
Package facts
| License | Apache-2.0 permissive |
| Python support | Supports the current Python release >=3.10 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 6 packagesapache-airflowapache-airflow-providers-common-compatpyspark-clientgrpcio-statusrequeststenacity |
| Maintenance | Actively maintained 6 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 1,187,469 / month, #4,246 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 5 - Production/StableEnvironment :: ConsoleEnvironment :: Web EnvironmentFramework :: Apache AirflowFramework :: Apache Airflow :: ProviderIntended Audience :: DevelopersIntended Audience :: System AdministratorsProgramming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Topic :: System :: Monitoring |
Evidence: apache_airflow_providers_apache_spark-6.3.1-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “airflow spark integration”
- apache-airflow-providers-apache-sparkIntegrates Apache Spark with Apache Airflow, enabling you to…
- apache-airflow-providers-apache-livyIntegrates Apache Livy with Apache Airflow to submit and monitor…
- apache-airflow-providers-microsoft-fabricProvides Apache Airflow operators to trigger, monitor, and…
Give your agent the search over MCP, or paste the wish link into any chat.
More Monitoring packages
Wraps any iterable to display a real-time progress bar in the terminal or Jupyter notebook, showing iteration count, elapsed time, and estimated time remaining.
Provides generated Python code for OpenTelemetry semantic conventions, enabling standardized attribute naming and constant definitions for instrumentation and telemetry collection.
Install it if you are using OpenTelemetry and want to follow semantic conventions correctly.
Provides the reference implementation of the OpenTelemetry API for collecting and exporting traces, metrics, and logs from Python applications.
Provides the abstract API and interfaces for OpenTelemetry instrumentation in Python, defining how to emit traces, metrics, and logs without tying code to a specific SDK implementation.
Exports OpenTelemetry observability data to an OpenTelemetry Collector using Protobuf-encoded messages over HTTP.
Install it if you are using OpenTelemetry in Python and need to send data to a Collector over HTTP.
Provides automatic instrumentation commands and programmatic APIs to inject distributed tracing into Python applications without code changes, detecting and instrumenting packages used by your program.
Install it if you need distributed tracing without code changes and have compatible instrumented packages in your environment.
See also apache-airflow-providers-apache-flink · apache-airflow-providers-celery · apache-airflow-providers-dbt-cloud · apache-airflow-providers-airbyte · apache-airflow-providers-cncf-kubernetes · apache-airflow-providers-databricks · apache-airflow-providers-jenkins · apache-airflow-providers-tableau · apache-airflow-providers-standard · apache-airflow-providers-microsoft-fabric