acryl-datahub-airflow-plugin

What it is and what it does

This package is a plugin for Apache Airflow that automatically extracts and forwards pipeline metadata to DataHub. It captures DAG structure, task inputs and outputs, and execution history without requiring code changes to your existing DAGs. The plugin integrates with Airflow's OpenLineage framework and enhances it with DataHub's SQL parser to extract column-level lineage from SQL operators, providing richer metadata than standard OpenLineage extractors alone.

The plugin supports multiple emission paths (REST, Kafka, or file) and activates automatically once configured in Airflow's configuration file with a DataHub connection. It's designed for teams running Airflow who want centralized visibility into data pipelines and lineage without manual instrumentation, making it useful for data governance, impact analysis, and compliance tracking.

Use it for:

• Automatically track dataset and column-level lineage from SQL-based Airflow tasks into a centralized DataHub catalog
• Monitor Airflow task execution status and duration to correlate failures with downstream data quality issues
• Enhance OpenLineage metadata with DataHub's advanced SQL parsing for more detailed lineage than standard extractors provide
• Stream Airflow pipeline metadata to Kafka for real-time consumption by other data governance tools
• Resolve temporary tables across multi-statement SQL tasks to capture complete lineage in complex ETL workflows

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Automatically captures pipeline lineage, task metadata, and run history from Apache Airflow DAGs and sends them to DataHub without requiring manual instrumentation.

Yes, if you run Apache Airflow 3.0+ and use DataHub for metadata management. The plugin is actively maintained, has no known vulnerabilities, low install friction, and solves the common problem of capturing lineage from Airflow without manual instrumentation. The permissive Apache-2.0 license poses no restrictions. Install only if you have a DataHub instance to send metadata to; it is not useful standalone.

Install

acryl-datahub-airflow-plugin on PyPI

pip install acryl-datahub-airflow-plugin

uv add acryl-datahub-airflow-plugin

poetry add acryl-datahub-airflow-plugin

Installing acryl-datahub-airflow-plugin

Before you install

Low install friction with a wheel distribution. Actively maintained—released 3 days ago with recent commits. Requires Apache Airflow and acryl-datahub as core dependencies, plus pydantic and apache-airflow-providers-openlineage.

License in practice

Licensed under Apache-2.0 (permissive), allowing commercial use, modification, and distribution with minimal restrictions.

Quickstart

pip install acryl-datahub-airflow-plugin

# Add to airflow.cfg:
# [datahub]
# enabled = True
# conn_id = datahub_rest_default

# Set up connection:
# airflow connections add datahub_rest_default --conn-type HTTP --conn-host http://localhost:8080

# Plugin activates automatically on Airflow startup

Requires Apache Airflow 3.0+ (version 1.6.0 or earlier for Airflow 2.x). Requires Python >= 3.10.

Verify before relying

• Whether column-level lineage extraction works with all SQL dialects or only specific ones
• Performance impact when enable_multi_statement_sql_parsing is enabled on large DAGs
• Whether the plugin works with all Airflow executor types or has known limitations

Package facts

Evidence: acryl_datahub_airflow_plugin-1.7.0.3-py3-none-any.whl

Tags

More Software Development packages