Apache Airflow Orchestration

Master production-grade Airflow DAG design with patterns for operators, sensors, and testing. Learn idempotent task design, dependency structures, and deployment strategies to orchestrate reliable data pipelines.

This skill generates Apache Airflow DAGs tailored for construction data workflows, handling everything from BIM model processing to validation and cost reporting. It provides a builder interface for composing tasks—bash commands, Python functions, file sensors, and branching logic—then outputs production-ready DAG code. Pre-built templates accelerate common patterns like BIM validation pipelines.

airflow-expert provides deep knowledge for building reliable, scalable data pipelines with Airflow. Learn DAG fundamentals, task dependencies, branching logic, dynamic task generation, and production operations patterns to orchestrate complex workflows.

Build production ML pipelines that handle data ingestion, model training, and deployment with Airflow DAGs, Kubeflow containers, and MLflow experiment tracking. This skill covers orchestration patterns, scheduling retraining jobs, managing model versions, and troubleshooting common workflow failures.

ML Pipeline Automation handles the complete machine learning lifecycle, automating data collection, preprocessing, model training, validation, and production deployment. Built on orchestration platforms like Airflow and Kubeflow, it ensures reproducible and scalable workflows with integrated monitoring and performance tracking.

MLOps Workflows guides you through production-grade machine learning operations using MLflow, covering experiment tracking, model registry management, deployment patterns, and monitoring. Learn to version models, run A/B tests, and implement CI/CD practices for reliable ML systems.