{"enrichment":{"faq":[{"a":"airflow-expert teaches you to create DAGs by defining a directed acyclic graph of tasks using the DAG class and decorators. Start with a DAG object specifying owner, schedule_interval, and start_date, then add tasks using operators like PythonOperator or BashOperator. The TaskFlow API (@dag and @task decorators) simplifies DAG creation by automatically handling dependencies through function calls, making your pipeline code more intuitive and Pythonic.","q":"How do I create Airflow DAGs?"},{"a":"airflow-expert covers task dependencies using set_upstream/set_downstream methods or the >> operator to define execution order. Scheduling is controlled by the schedule_interval parameter (cron expressions or timedelta objects). Dependencies ensure tasks run in sequence or parallel as intended, while scheduling determines when DAG runs trigger. Understanding both prevents race conditions and ensures predictable workflow execution across your data pipeline.","q":"What are Airflow task dependencies and how do scheduling work?"},{"a":"airflow-expert explains that sensors are operators that wait for a condition to be met before proceeding. Common sensors include FileSensor (waits for files), ExternalTaskSensor (waits for other DAG tasks), and S3KeySensor (monitors S3 buckets). Configure sensors with poke_interval and timeout parameters to control polling frequency and maximum wait time. Sensors enable event-driven pipelines where downstream tasks only execute once dependencies are satisfied.","q":"How do I use Airflow sensors in my workflows?"},{"a":"airflow-expert demonstrates XCom (cross-communication) as Airflow's mechanism for passing data between tasks. Use task.xcom_push(key, value) to send data and task.xcom_pull(task_ids, key) to retrieve it. The TaskFlow API automatically handles XCom by allowing task functions to return values that downstream tasks receive as parameters. XCom enables dynamic workflows where task outputs drive subsequent task behavior, though large data transfers should use external storage.","q":"What is XCom communication between tasks in Airflow?"},{"a":"airflow-expert covers error handling through task retries, SLA monitoring, and exception handling. Set retries and retry_delay on operators to automatically retry failed tasks. Use on_failure_callback and on_retry_callback to execute custom logic when tasks fail or retry. Implement idempotent tasks to safely handle re-execution. Configure SLAs to alert when tasks exceed time thresholds. Production pipelines combine these patterns with alerting integrations to ensure reliability and observability.","q":"How do I implement error handling and retry logic in Airflow?"},{"a":"airflow-expert emphasizes storing sensitive data in Airflow Connections and Variables rather than hardcoding credentials. Use role-based access control (RBAC) to restrict UI and API access. Implement idempotent DAGs for safe reruns and backfills. Monitor DAG performance with logging and metrics. Separate development, staging, and production environments. Version control your DAGs and test them locally before deployment. Apply pool configuration to limit concurrent task execution and prevent resource exhaustion in your infrastructure.","q":"What are Airflow production best practices and security considerations?"}],"shadow_tags":["workflow-orchestration","etl-pipelines","task-scheduling","data-engineering","pipeline-patterns","distributed-computing","fault-tolerance","monitoring-alerts","infrastructure-automation"],"summary_rewrite":"airflow-expert provides deep knowledge for building reliable, scalable data pipelines with Airflow. Learn DAG fundamentals, task dependencies, branching logic, dynamic task generation, and production operations patterns to orchestrate complex workflows."},"files":[{"bytes":22915,"path":"stdlib/data/airflow-expert/SKILL.md","sha256":"80e030575067feeb1f7a843c5c045b56ce2f7d110a090909794ca93746991e0e","url":"https://skillfed.io/files/personamanagmentlayer/pcl/airflow-expert/6ccea02e/SKILL.md"}],"id":"personamanagmentlayer/pcl/airflow-expert","links":{"html":"https://skillfed.io/personamanagmentlayer/pcl/airflow-expert","md":"https://skillfed.io/personamanagmentlayer/pcl/airflow-expert.md","repo":"https://github.com/personamanagmentlayer/pcl"},"meta":{"agents_supported":[],"first_seen":"2026-07-28","forks":8,"language":"TypeScript","last_updated":"2026-03-30","license":"Apache-2.0","name":"airflow-expert","publisher":"personamanagmentlayer","stars":38},"relations":{"similar":[{"id":"Aradotso/data-skills/apache-airflow-orchestration"},{"id":"manutej/luxor-claude-marketplace/apache-airflow-orchestration"},{"id":"wshobson/agents/airflow-dag-patterns"},{"id":"pluginagentmarketplace/custom-plugin-data-engineer/etl-tools"},{"id":"datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction/airflow-dag"},{"id":"ancoleman/ai-design-components/transforming-data"},{"id":"secondsky/claude-skills/ml-pipeline-automation"},{"id":"astronomer/agents/cosmos-dbt-core"},{"id":"Aradotso/data-skills/data-engineering-medallion-pipeline"},{"id":"Aradotso/data-skills/google-cloud-data-engineering-hub"}]},"slug":{"owner":"personamanagmentlayer","repo":"pcl","skill":"airflow-expert"},"version":"6ccea02e"}