apache-airflow-providers-apache-hdfs
Provider package apache-airflow-providers-apache-hdfs for Apache Airflow
Decision gist · record as of 2026-08-14
Yes, if you run Apache Airflow and need to orchestrate tasks on a Hadoop cluster. The package is actively maintained, has no known vulnerabilities, installs with low friction, and is part of the official Airflow provider ecosystem. Install it only if you have an HDFS cluster to connect to; it is not useful as a standalone tool.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires an existing Apache Airflow installation (>=2.11.0) and network connectivity to an HDFS cluster or WebHDFS endpoint.
- Low friction install with a pure-wheel distribution.
- Actively maintained as of 6 days ago with recent commits; part of the Apache Airflow ecosystem with 46490 repository stars.
License · maintenance · safety
Apache-2.0 (permissive) — Licensed under Apache-2.0 (permissive), allowing use in commercial and open-source projects with minimal restrictions beyond attribution and liability disclaimers.
last release 2026-08-08 (6 days) · last repo commit 2026-08-14 · 46,490 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 249,279 downloads/mo, #8,652 on PyPI
Alternatives
Verify before relying
pip install apache-airflow-providers-apache-hdfs
from airflow.providers.apache.hdfs.operators.hdfs import HdfsOperator
from airflow import DAG
with DAG('hdfs_example') as dag:
task = HdfsOperator(task_id='read_hdfs', ...)- Specific operators, hooks, or sensors provided beyond generic HDFS file operations.
- Support for Kerberos authentication and other security mechanisms in HDFS connections.
- Data serialization format support (Avro, Parquet) beyond what fastavro and pandas provide.
What it is and what it does
This is an Apache Airflow provider package that adds HDFS and WebHDFS connectivity to Airflow workflows. It extends Airflow with operators, hooks, and sensors for interacting with Hadoop Distributed File System clusters, allowing you to read, write, and manage files as part of data pipelines. The package depends on apache-airflow, the hdfs client library, fastavro for Avro serialization, and pandas for dataframe operations.
The package is designed for teams running Airflow orchestration on top of Hadoop infrastructure. It abstracts away the complexity of HDFS client configuration and integrates directly into Airflow's task execution model, so you can define file operations declaratively in your DAGs. It supports Python 3.10 through 3.14 and is actively maintained by the Apache Airflow project.
Use it for
- Orchestrate ETL pipelines that read raw data from HDFS, transform it, and write results back to the cluster.
- Monitor HDFS file availability as a sensor in Airflow DAGs before triggering downstream processing tasks.
- Integrate Hadoop-based data lakes with Airflow workflows for scheduled batch processing and data movement.
- Build data validation tasks that check file existence, size, or format on HDFS before proceeding with pipeline stages.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you run Apache Airflow and need to orchestrate tasks on a Hadoop cluster.
The package is actively maintained, has no known vulnerabilities, installs with low friction, and is part of the official Airflow provider ecosystem. Install it only if you have an HDFS cluster to connect to; it is not useful as a standalone tool.
Install
apache-airflow-providers-apache-hdfs on PyPI
Before you install
Low friction install with a pure-wheel distribution. Actively maintained as of 6 days ago with recent commits; part of the Apache Airflow ecosystem with 46490 repository stars. Requires Apache Airflow >=2.11.0 and Python >=3.10.
Requires an existing Apache Airflow installation (>=2.11.0) and network connectivity to an HDFS cluster or WebHDFS endpoint.
License in practice
Licensed under Apache-2.0 (permissive), allowing use in commercial and open-source projects with minimal restrictions beyond attribution and liability disclaimers.
Quickstart
pip install apache-airflow-providers-apache-hdfs
from airflow.providers.apache.hdfs.operators.hdfs import HdfsOperator
from airflow import DAG
with DAG('hdfs_example') as dag:
task = HdfsOperator(task_id='read_hdfs', ...)Verify before relying
- Specific operators, hooks, or sensors provided beyond generic HDFS file operations.
- Support for Kerberos authentication and other security mechanisms in HDFS connections.
- Data serialization format support (Avro, Parquet) beyond what fastavro and pandas provide.
Package facts
| License | Apache-2.0 permissive |
| Python support | Supports the current Python release >=3.10 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 5 packagesapache-airflowapache-airflow-providers-common-compathdfsfastavropandas |
| Maintenance | Actively maintained 6 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 249,279 / month, #8,652 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 5 - Production/StableEnvironment :: ConsoleEnvironment :: Web EnvironmentFramework :: Apache AirflowFramework :: Apache Airflow :: ProviderIntended Audience :: DevelopersIntended Audience :: System AdministratorsProgramming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Topic :: System :: Monitoring |
Evidence: apache_airflow_providers_apache_hdfs-4.12.2-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “airflow hdfs integration”
- apache-airflow-providers-apache-hdfsIntegrates Apache Hadoop Distributed File System (HDFS) and WebHDFS…
- snakebite-py3Provides a pure Python HDFS client library and command-line interface…
- hdfsHdfsCLI provides Python bindings and a command-line interface for…
Give your agent the search over MCP, or paste the wish link into any chat.
More Monitoring packages
Wraps any iterable to display a real-time progress bar in the terminal or Jupyter notebook, showing iteration count, elapsed time, and estimated time remaining.
Provides generated Python code for OpenTelemetry semantic conventions, enabling standardized attribute naming and constant definitions for instrumentation and telemetry collection.
Install it if you are using OpenTelemetry and want to follow semantic conventions correctly.
Provides the reference implementation of the OpenTelemetry API for collecting and exporting traces, metrics, and logs from Python applications.
Provides the abstract API and interfaces for OpenTelemetry instrumentation in Python, defining how to emit traces, metrics, and logs without tying code to a specific SDK implementation.
Exports OpenTelemetry observability data to an OpenTelemetry Collector using Protobuf-encoded messages over HTTP.
Install it if you are using OpenTelemetry in Python and need to send data to a Collector over HTTP.
Provides automatic instrumentation commands and programmatic APIs to inject distributed tracing into Python applications without code changes, detecting and instrumenting packages used by your program.
Install it if you need distributed tracing without code changes and have compatible instrumented packages in your environment.
See also apache-airflow-providers-salesforce · apache-airflow-providers-databricks · hdfs · apache-airflow-providers-apache-pig · snakebite-py3 · apache-airflow-providers-papermill · apache-airflow-providers-apache-hive · apache-airflow-providers-opensearch · apache-airflow-providers-cohere · apache-airflow-providers-common-io