apache-airflow-providers-apache-hdfs

What it is and what it does

This is an Apache Airflow provider package that adds HDFS and WebHDFS connectivity to Airflow workflows. It extends Airflow with operators, hooks, and sensors for interacting with Hadoop Distributed File System clusters, allowing you to read, write, and manage files as part of data pipelines. The package depends on apache-airflow, the hdfs client library, fastavro for Avro serialization, and pandas for dataframe operations.

The package is designed for teams running Airflow orchestration on top of Hadoop infrastructure. It abstracts away the complexity of HDFS client configuration and integrates directly into Airflow's task execution model, so you can define file operations declaratively in your DAGs. It supports Python 3.10 through 3.14 and is actively maintained by the Apache Airflow project.

Use it for:

• Orchestrate ETL pipelines that read raw data from HDFS, transform it, and write results back to the cluster.
• Monitor HDFS file availability as a sensor in Airflow DAGs before triggering downstream processing tasks.
• Integrate Hadoop-based data lakes with Airflow workflows for scheduled batch processing and data movement.
• Build data validation tasks that check file existence, size, or format on HDFS before proceeding with pipeline stages.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Integrates Apache Hadoop Distributed File System (HDFS) and WebHDFS with Apache Airflow, enabling workflows to read, write, and manage files on Hadoop clusters.

Yes, if you run Apache Airflow and need to orchestrate tasks on a Hadoop cluster. The package is actively maintained, has no known vulnerabilities, installs with low friction, and is part of the official Airflow provider ecosystem. Install it only if you have an HDFS cluster to connect to; it is not useful as a standalone tool.

Install

apache-airflow-providers-apache-hdfs on PyPI

pip install apache-airflow-providers-apache-hdfs

uv add apache-airflow-providers-apache-hdfs

poetry add apache-airflow-providers-apache-hdfs

Installing apache-airflow-providers-apache-hdfs

Before you install

Low friction install with a pure-wheel distribution. Actively maintained as of 6 days ago with recent commits; part of the Apache Airflow ecosystem with 46490 repository stars. Requires Apache Airflow >=2.11.0 and Python >=3.10.

License in practice

Licensed under Apache-2.0 (permissive), allowing use in commercial and open-source projects with minimal restrictions beyond attribution and liability disclaimers.

Quickstart

pip install apache-airflow-providers-apache-hdfs

from airflow.providers.apache.hdfs.operators.hdfs import HdfsOperator
from airflow import DAG

with DAG('hdfs_example') as dag:
    task = HdfsOperator(task_id='read_hdfs', ...)

Requires an existing Apache Airflow installation (>=2.11.0) and network connectivity to an HDFS cluster or WebHDFS endpoint.

Verify before relying

• Specific operators, hooks, or sensors provided beyond generic HDFS file operations.
• Support for Kerberos authentication and other security mechanisms in HDFS connections.
• Data serialization format support (Avro, Parquet) beyond what fastavro and pandas provide.

Package facts

Evidence: apache_airflow_providers_apache_hdfs-4.12.2-py3-none-any.whl

Keywords: airflow-provider, apache.hdfs, airflow, integration

Tags

More Monitoring packages