apache-airflow-providers-apache-kafka

What it is and what it does

This is an Apache Airflow provider package that extends Airflow with Kafka integration capabilities. It wraps the confluent-kafka library and exposes Kafka operators and hooks within Airflow's DAG framework, allowing you to define tasks that produce messages to or consume messages from Kafka topics as part of orchestrated workflows.

The package is maintained as part of the Apache Airflow project and is actively developed. It supports Python 3.10, 3.11, 3.12, 3.13, and 3.14, and requires Airflow >=2.11.0. Installation is straightforward via pip, and it has low dependency friction—the main runtime dependencies are apache-airflow, confluent-kafka, asgiref, and apache-airflow-providers-common-compat. No known security vulnerabilities are recorded.

Use it for:

• Stream event data into Kafka topics as part of a scheduled Airflow pipeline.
• Consume Kafka messages in an Airflow task and process them (e.g., transform, validate, load into a data warehouse).
• Orchestrate multi-stage data pipelines where Kafka serves as the message broker between Airflow tasks.
• Integrate Kafka-based microservices with Airflow workflows for event-driven data processing.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Integrates Apache Kafka with Apache Airflow, providing operators and hooks to produce and consume messages from Kafka topics as part of Airflow workflows.

Yes. This is a stable, actively maintained provider package from the Apache Airflow project with no known vulnerabilities, low install friction, and a permissive Apache-2.0 license. Install it if you need to integrate Kafka into Airflow DAGs. The only prerequisite is an existing Airflow >=2.11.0 installation and a Kafka broker to connect to.

Install

apache-airflow-providers-apache-kafka on PyPI

pip install apache-airflow-providers-apache-kafka

uv add apache-airflow-providers-apache-kafka

poetry add apache-airflow-providers-apache-kafka

Installing apache-airflow-providers-apache-kafka

Before you install

Low friction installation. Active maintenance with a release 6 days ago. Requires Apache Airflow >=2.11.0 and confluent-kafka >=2.6.0 (or >=2.13.2 for Python 3.14+), both straightforward dependencies.

License in practice

Apache-2.0 permissive license. You can use, modify, and distribute this package freely in commercial and private projects, provided you include a copy of the license and any NOTICE files.

Quickstart

pip install apache-airflow-providers-apache-kafka

from airflow.providers.apache.kafka.operators.produce_to_topic import ProduceToTopicOperator

task = ProduceToTopicOperator(
    task_id='produce_to_kafka',
    topic='my_topic',
    producer_config={'bootstrap.servers': 'kafka_broker'}
)

Requires an existing Apache Airflow installation (>=2.11.0) and a running Kafka broker accessible at the configured bootstrap.servers address.

Verify before relying

• Specific operators and hooks available beyond produce/consume (e.g., consumer groups, offset management, schema registry integration).
• Whether optional extras (google, msk, common.messaging) are commonly needed or niche use cases.
• Example producer_config parameters and typical Kafka broker connection patterns.

Package facts

Evidence: apache_airflow_providers_apache_kafka-1.15.1-py3-none-any.whl

Keywords: airflow-provider, apache.kafka, airflow, integration

Tags

More Monitoring packages