airflow-dag-patterns
Master production-grade Airflow DAG design with patterns for operators, sensors, and testing. Learn idempotent task design, dependency structures, and deployment strategies to orchestrate reliable data pipelines.
Airflow DAG Patterns teaches production-ready Apache Airflow design with best practices for operators, sensors, and testing.
AI-generated summary based on this skill's SKILL.md
Install
wshobson/agents/airflow-dag-patterns · repository language: Python
git clone https://github.com/wshobson/agents
cp -r agents/plugins/data-engineering/skills/airflow-dag-patterns ~/.claude/skills/airflow-dag-patternsnpx skillfed install wshobson/agents/airflow-dag-patternsFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How to build Apache Airflow DAGs with best practices?
airflow-dag-patterns teaches you to construct production-ready DAGs using proven design patterns. The skill covers structuring tasks with proper dependencies, implementing idempotent workflows, configuring retry logic and timeouts, and organizing code for maintainability. You'll learn to leverage the TaskFlow API, define clear task boundaries, and follow Airflow conventions for reliable orchestration.
What are airflow dag best practices for production deployment?
airflow-dag-patterns emphasizes production readiness through idempotent task design, robust error handling, and deployment strategies. Key practices include setting appropriate retry policies, using sensors effectively, structuring dependencies to prevent cascading failures, testing DAGs locally before production, and monitoring failed runs. The skill guides you through configuration patterns that ensure your workflows remain reliable at scale.
How do you create a data pipeline with Airflow?
airflow-dag-patterns walks you through designing and orchestrating complete data pipelines. You'll learn to define task sequences, connect operators for data extraction and transformation, use sensors to wait for external conditions, and structure XCom communication between tasks. The skill covers real ETL patterns, batch job scheduling, and how to compose operators and sensors into cohesive workflows.
What airflow operators and sensors should I implement?
airflow-dag-patterns teaches custom operator and sensor implementation alongside built-in options. You'll learn when to use standard operators versus building custom ones, how sensors reschedule or poke for readiness, and patterns for task communication. The skill covers sensor modes, operator inheritance, and practical examples of extending Airflow's capabilities for your specific pipeline needs.
How do you test and debug Airflow DAGs locally?
airflow-dag-patterns provides testing strategies for both local development and production environments. You'll learn to validate DAG structure, test individual tasks, simulate scheduling behavior, and debug failed runs. The skill covers tools for local testing, logging practices, and techniques to identify issues before deployment, ensuring your workflows behave correctly across environments.
How should task dependencies and idempotent workflows be configured?
airflow-dag-patterns covers dependency setup patterns that prevent race conditions and ensure correct execution order. You'll learn idempotent task design principles—making tasks safe to retry without side effects—and configuring catchup settings, retry logic, and timeouts. The skill teaches how to structure workflows so failures don't corrupt data and retries succeed reliably.
SKILL.md
rendered from the published skill — quoted content, verbatim
Apache Airflow DAG Patterns
Production-ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies.
When to Use This Skill
- Creating data pipeline orchestration with Airflow
- Designing DAG structures and dependencies
- Implementing custom operators and sensors
- Testing Airflow DAGs locally
- Setting up Airflow in production
- Debugging failed DAG runs
Core Concepts
1. DAG Design Principles
| Principle | Description |
|---|---|
| Idempotent | Running twice produces same result |
| Atomic | Tasks succeed or fail completely |
| Incremental | Process only new/changed data |
| Observable | Logs, metrics, alerts at every step |
2. Task Dependencies
```python
Linear
task1 >> task2 >> task3
Fan-out
task1 >> [task2, task3, task4]
Fan-in
[task1, task2, task3] >>
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 2 files
plugins/data-engineering/skills/airflow-dag-patterns/SKILL.md
plugins/data-engineering/skills/airflow-dag-patterns/references/details.md