apache-airflow-core

What it is and what it does

Apache Airflow Core is the runtime engine for Apache Airflow, a platform for authoring, scheduling, and monitoring workflows. It provides the scheduler that reads DAG definitions and triggers tasks on schedule, the REST API server for external interaction, the DAG file processor that parses workflow definitions, and the triggerer component for event-driven task execution. It is designed for data engineers and DevOps teams building complex, multi-step data pipelines that may span multiple systems and require visibility into execution state.

The package depends on a large ecosystem of providers and utilities—fastapi for the web server, SQLAlchemy-based ORM layers for metadata storage, jinja2 for templating, cryptography for secure credential handling, and cron scheduling libraries. It targets modern Python versions and is classified as production-stable. Installation is straightforward from a wheel, though running Airflow in practice requires external infrastructure: a database to store DAG metadata and execution history, and typically a message broker or Kubernetes cluster for distributed task execution.

Use it for:

• Schedule and execute recurring ETL jobs that extract data from multiple sources, transform it, and load it into a data warehouse.
• Orchestrate machine learning pipelines with dependencies between data preparation, model training, and evaluation stages.
• Monitor and retry failed batch jobs across distributed systems with automatic alerting and logging.
• Coordinate multi-team workflows where different teams own different pipeline stages and need visibility into upstream/downstream dependencies.
• Build event-driven data ingestion pipelines that react to external triggers and fan out work across worker nodes.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Apache Airflow's core runtime that schedules, executes, and monitors data pipelines defined as directed acyclic graphs (DAGs), with a REST API server and task triggering engine.

Yes. Apache Airflow Core is the foundation of a mature, widely-adopted orchestration platform with 46489 repository stars and monthly downloads in the millions. It is actively maintained, permissively licensed, and production-stable. Install it if you need to build or run Airflow workflows; it is a prerequisite for any Airflow deployment. Be aware that it brings 65 dependencies and requires external infrastructure to operate at scale.

Install

apache-airflow-core on PyPI

pip install apache-airflow-core

uv add apache-airflow-core

poetry add apache-airflow-core

Installing apache-airflow-core

Before you install

Active maintenance with a release 2 days old and 46489 repository stars. Low install friction from a pure-wheel distribution. Depends on 65 runtime packages including fastapi, cryptography, and database/scheduling libraries—a substantial but standard stack for a production orchestration platform.

License in practice

Apache License 2.0 (permissive) allows commercial use, modification, and redistribution with minimal restrictions, making it suitable for enterprise and proprietary workflows.

Quickstart

pip install apache-airflow-core

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG('example', start_date=datetime(2025, 1, 1)) as dag:
    task = BashOperator(task_id='hello', bash_command='echo hello')

Requires Python 3.10 or later (3.15 excluded); a running database backend and a message broker for distributed execution are typical production requirements.

Verify before relying

• Specific performance characteristics or throughput limits under typical workloads
• Whether all 65 runtime dependencies are strictly required or some are optional for minimal setups
• Compatibility details with specific database and message broker versions

Package facts

Evidence: apache_airflow_core-3.3.1-py3-none-any.whl

Keywords: airflow, automation, dag, data, orchestration, pipelines, workflow

Tags

More Monitoring packages