airflow-expert
airflow-expert provides deep knowledge for building reliable, scalable data pipelines with Airflow. Learn DAG fundamentals, task dependencies, branching logic, dynamic task generation, and production operations patterns to orchestrate complex workflows.
airflow-expert helps you design and build complex data pipelines using Apache Airflow DAGs, operators, and task orchestration.
AI-generated summary based on this skill's SKILL.md
Decision gist · record as of 2026-03-30
airflow-expert helps you design and build complex data pipelines using Apache Airflow DAGs, operators, and task orchestration. airflow-expert provides deep knowledge for building reliable, scalable data pipelines with Airflow. Learn DAG fundamentals, task dependencies, branching logic, dynamic task generation, and production operations patterns to orchestrate complex workflows.
Use it when
- airflow-expert covers task dependencies using set_upstream/set_downstream methods or the >> operator to define execution order.
- airflow-expert explains that sensors are operators that wait for a condition to be met before proceeding.
Verify before relying
Read SKILL.md below before installing (1 file). Open directory: indexed for reading, not audited.
Install
personamanagmentlayer/pcl/airflow-expert · repository language: TypeScript
Open directory. Skills are indexed for reading, not audited. Review a skill's body before installing it.
Frequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I create Airflow DAGs?
airflow-expert teaches you to create DAGs by defining a directed acyclic graph of tasks using the DAG class and decorators. Start with a DAG object specifying owner, schedule_interval, and start_date, then add tasks using operators like PythonOperator or BashOperator. The TaskFlow API (@dag and @task decorators) simplifies DAG creation by automatically handling dependencies through function calls, making your pipeline code more intuitive and Pythonic.
What are Airflow task dependencies and how do scheduling work?
airflow-expert covers task dependencies using set_upstream/set_downstream methods or the >> operator to define execution order. Scheduling is controlled by the schedule_interval parameter (cron expressions or timedelta objects). Dependencies ensure tasks run in sequence or parallel as intended, while scheduling determines when DAG runs trigger. Understanding both prevents race conditions and ensures predictable workflow execution across your data pipeline.
How do I use Airflow sensors in my workflows?
airflow-expert explains that sensors are operators that wait for a condition to be met before proceeding. Common sensors include FileSensor (waits for files), ExternalTaskSensor (waits for other DAG tasks), and S3KeySensor (monitors S3 buckets). Configure sensors with poke_interval and timeout parameters to control polling frequency and maximum wait time. Sensors enable event-driven pipelines where downstream tasks only execute once dependencies are satisfied.
What is XCom communication between tasks in Airflow?
airflow-expert demonstrates XCom (cross-communication) as Airflow's mechanism for passing data between tasks. Use task.xcom_push(key, value) to send data and task.xcom_pull(task_ids, key) to retrieve it. The TaskFlow API automatically handles XCom by allowing task functions to return values that downstream tasks receive as parameters. XCom enables dynamic workflows where task outputs drive subsequent task behavior, though large data transfers should use external storage.
How do I implement error handling and retry logic in Airflow?
airflow-expert covers error handling through task retries, SLA monitoring, and exception handling. Set retries and retry_delay on operators to automatically retry failed tasks. Use on_failure_callback and on_retry_callback to execute custom logic when tasks fail or retry. Implement idempotent tasks to safely handle re-execution. Configure SLAs to alert when tasks exceed time thresholds. Production pipelines combine these patterns with alerting integrations to ensure reliability and observability.
What are Airflow production best practices and security considerations?
airflow-expert emphasizes storing sensitive data in Airflow Connections and Variables rather than hardcoding credentials. Use role-based access control (RBAC) to restrict UI and API access. Implement idempotent DAGs for safe reruns and backfills. Monitor DAG performance with logging and metrics. Separate development, staging, and production environments. Version control your DAGs and test them locally before deployment. Apply pool configuration to limit concurrent task execution and prevent resource exhaustion in your infrastructure.
SKILL.md
Rendered from the published skill. Quoted content, verbatim.
Apache Airflow Expert
You are an expert in Apache Airflow with deep knowledge of DAG design, task orchestration, operators, sensors, XComs, dynamic task generation, and production operations. You design and manage complex data pipelines that are reliable, maintainable, and scalable.
Core Expertise
DAG Fundamentals
Basic DAG Structure: ```python from airflow import DAG from airflow.operators.python import PythonOperator from airflow.operators.bash import BashOperator from datetime import datetime, timedelta
Default arguments
default_args = { 'owner': 'data-engineering', 'depends_on_past': False, 'email': ['alerts@company.com'], 'email_on_failure': True, 'email_on_retry': False, 'retries': 3, 'retry_delay':
(truncated - see the full file via the links below)
File tree — 1 file
stdlib/data/airflow-expert/SKILL.md
Let your AI agent find skills like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 56,283 agent skills by what they can do, searchable in plain language.
wish › “Design and build complex data pipelines with Apache Airflow DAGs”
Give your agent the search over MCP, or paste the wish link into any chat. No install? Search from any chat →
Related skills
This skill generates Apache Airflow DAGs tailored for construction data workflows, handling everything from BIM model processing to validation and cost reporting. It provides a builder interface for composing tasks—bash commands, Python functions, file sensors, and branching logic—then outputs production-ready DAG code. Pre-built templates accelerate common patterns like BIM validation pipelines.
Learn to architect and deploy data pipelines with Apache Airflow, covering DAG design, operators, task dependencies, branching logic, and scheduling patterns. This skill guides you through building scalable workflows from basic structures to advanced patterns like dynamic task generation and conditional execution. Deploy production-grade orchestration across single machines or distributed clusters.
Master production-grade Airflow DAG design with patterns for operators, sensors, and testing. Learn idempotent task design, dependency structures, and deployment strategies to orchestrate reliable data pipelines.
Build production ML pipelines that handle data ingestion, model training, and deployment with Airflow DAGs, Kubeflow containers, and MLflow experiment tracking. This skill covers orchestration patterns, scheduling retraining jobs, managing model versions, and troubleshooting common workflow failures.
ML Pipeline Automation handles the complete machine learning lifecycle, automating data collection, preprocessing, model training, validation, and production deployment. Built on orchestration platforms like Airflow and Kubeflow, it ensures reproducible and scalable workflows with integrated monitoring and performance tracking.
Cloudflare Expert provides in-depth knowledge for building serverless applications on Cloudflare's edge platform. Learn to deploy code globally with Workers, optimize caching and CDN performance, implement security features like WAF and DDoS protection, and manage distributed state using Workers KV and Durable Objects.
More skills microservices-expert (Apache-2.0)