apache-airflow-providers-apache-beam

What it is and what it does

This is an Apache Airflow provider package that bridges Apache Beam into Airflow's workflow orchestration model. It supplies operators and hooks that allow you to define and schedule Beam pipelines as tasks within Airflow DAGs, treating data processing jobs as first-class workflow components. The package depends on apache-airflow, apache-beam, pyarrow, and numpy, and is designed for teams already running Airflow who want to orchestrate Beam workloads without leaving the Airflow ecosystem.

The package is actively maintained, supports Python 3.10–3.13, and requires Apache Airflow >=2.11.0 and Apache Beam >=2.69.0. It carries an Apache-2.0 license and integrates with optional Google Cloud dependencies for Dataflow support. Installation is straightforward via pip, and the package is positioned as a standard extension for Airflow users working with Beam-based data pipelines.

Use it for:

• Schedule and monitor Apache Beam batch jobs as part of a larger Airflow data pipeline.
• Orchestrate Beam streaming pipelines with Airflow's scheduling and retry logic.
• Integrate Beam processing with other Airflow operators in a unified DAG.
• Run Beam jobs on Google Cloud Dataflow through Airflow task dependencies.
• Manage dependencies between Beam pipelines and upstream/downstream data tasks.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Integrates Apache Beam data processing pipelines into Apache Airflow workflows, enabling orchestration of batch and streaming jobs through Airflow's DAG framework.

Yes, if you are already running Apache Airflow and need to orchestrate Apache Beam pipelines. The package is actively maintained, carries no known vulnerabilities, has low install friction, and integrates cleanly with Airflow's task model. Install only if you have both Airflow and Beam in your stack; it adds no value as a standalone library.

Install

apache-airflow-providers-apache-beam on PyPI

pip install apache-airflow-providers-apache-beam

uv add apache-airflow-providers-apache-beam

poetry add apache-airflow-providers-apache-beam

Installing apache-airflow-providers-apache-beam

Before you install

Low friction install as a pure Python wheel. Actively maintained with recent releases; requires Apache Airflow >=2.11.0 and Apache Beam >=2.69.0, plus pyarrow and numpy. Supports Python 3.10–3.13.

License in practice

Apache-2.0 permissive license allows commercial and private use without restriction, typical for Apache Foundation projects.

Quickstart

pip install apache-airflow-providers-apache-beam

from airflow.providers.apache.beam.operators.beam import BeamRunPythonPipelineOperator

task = BeamRunPythonPipelineOperator(
    task_id='beam_job',
    py_file='pipeline.py'
)

Requires an existing Apache Airflow installation (>=2.11.0) and Apache Beam (>=2.69.0) to be available in the environment.

Verify before relying

• Whether the provider supports all Apache Beam runners (Direct, Dataflow, Spark, Flink) or a subset.
• Whether optional GCP integration (apache-beam[gcp]) is required for Dataflow or only for specific features.
• Performance characteristics when orchestrating large-scale or long-running Beam pipelines.

Package facts

Evidence: apache_airflow_providers_apache_beam-6.2.3-py3-none-any.whl

Keywords: airflow-provider, apache.beam, airflow, integration

Tags

More Monitoring packages