airflow-expert
airflow-expert provides deep knowledge for building reliable, scalable data pipelines with Airflow. Learn DAG fundamentals, task dependencies, branching logic, dynamic task generation, and production operations patterns to orchestrate complex workflows.
airflow-expert helps you design and build complex data pipelines using Apache Airflow DAGs, operators, and task orchestration.
AI-generated summary based on this skill's SKILL.md
Install
personamanagmentlayer/pcl/airflow-expert · repository language: TypeScript
git clone https://github.com/personamanagmentlayer/pcl
cp -r pcl/stdlib/data/airflow-expert ~/.claude/skills/airflow-expertnpx skillfed install personamanagmentlayer/pcl/airflow-expertFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I create Airflow DAGs?
airflow-expert teaches you to create DAGs by defining a directed acyclic graph of tasks using the DAG class and decorators. Start with a DAG object specifying owner, schedule_interval, and start_date, then add tasks using operators like PythonOperator or BashOperator. The TaskFlow API (@dag and @task decorators) simplifies DAG creation by automatically handling dependencies through function calls, making your pipeline code more intuitive and Pythonic.
What are Airflow task dependencies and how do scheduling work?
airflow-expert covers task dependencies using set_upstream/set_downstream methods or the >> operator to define execution order. Scheduling is controlled by the schedule_interval parameter (cron expressions or timedelta objects). Dependencies ensure tasks run in sequence or parallel as intended, while scheduling determines when DAG runs trigger. Understanding both prevents race conditions and ensures predictable workflow execution across your data pipeline.
How do I use Airflow sensors in my workflows?
airflow-expert explains that sensors are operators that wait for a condition to be met before proceeding. Common sensors include FileSensor (waits for files), ExternalTaskSensor (waits for other DAG tasks), and S3KeySensor (monitors S3 buckets). Configure sensors with poke_interval and timeout parameters to control polling frequency and maximum wait time. Sensors enable event-driven pipelines where downstream tasks only execute once dependencies are satisfied.
What is XCom communication between tasks in Airflow?
airflow-expert demonstrates XCom (cross-communication) as Airflow's mechanism for passing data between tasks. Use task.xcom_push(key, value) to send data and task.xcom_pull(task_ids, key) to retrieve it. The TaskFlow API automatically handles XCom by allowing task functions to return values that downstream tasks receive as parameters. XCom enables dynamic workflows where task outputs drive subsequent task behavior, though large data transfers should use external storage.
How do I implement error handling and retry logic in Airflow?
airflow-expert covers error handling through task retries, SLA monitoring, and exception handling. Set retries and retry_delay on operators to automatically retry failed tasks. Use on_failure_callback and on_retry_callback to execute custom logic when tasks fail or retry. Implement idempotent tasks to safely handle re-execution. Configure SLAs to alert when tasks exceed time thresholds. Production pipelines combine these patterns with alerting integrations to ensure reliability and observability.
What are Airflow production best practices and security considerations?
airflow-expert emphasizes storing sensitive data in Airflow Connections and Variables rather than hardcoding credentials. Use role-based access control (RBAC) to restrict UI and API access. Implement idempotent DAGs for safe reruns and backfills. Monitor DAG performance with logging and metrics. Separate development, staging, and production environments. Version control your DAGs and test them locally before deployment. Apply pool configuration to limit concurrent task execution and prevent resource exhaustion in your infrastructure.
SKILL.md
rendered from the published skill — quoted content, verbatim
Apache Airflow Expert
You are an expert in Apache Airflow with deep knowledge of DAG design, task orchestration, operators, sensors, XComs, dynamic task generation, and production operations. You design and manage complex data pipelines that are reliable, maintainable, and scalable.
Core Expertise
DAG Fundamentals
Basic DAG Structure: ```python from airflow import DAG from airflow.operators.python import PythonOperator from airflow.operators.bash import BashOperator from datetime import datetime, timedelta
Default arguments
default_args = { 'owner': 'data-engineering', 'depends_on_past': False, 'email': ['alerts@company.com'], 'email_on_failure': True, 'email_on_retry': False, 'retries': 3, 'retry_delay':
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 1 file
stdlib/data/airflow-expert/SKILL.md