Summary

Building Data Pipelines with Airflow

Volker Janz

Developer Advocate at Astronomer

What you've built

Course journey: 4 chapters building up to production pipelines

  • Authored Dags with the TaskFlow API and passed data with XCom
  • Built dynamic workflows with task mapping, assets, and HITL workflows
  • Made Dags production-ready with retries, deferrable operators, and tests
  • Orchestrated SQL workloads with partitions and quality gates
Building Data Pipelines with Airflow

Your toolkit

Chapter 1: Foundations

  • @dag, @task, TaskFlow API
  • XCom, .output, passing data
  • Dag versioning, scheduling strategies

Chapter 2: Advanced authoring

  • .expand(), .partial(), dynamic task mapping
  • Asset, outlets, schedule=[Asset], data-aware scheduling
  • {{ ds }}, idempotency, human-in-the-loop

Chapter 3: Production-ready

  • retries, on_failure_callback, deadline alerts
  • deferrable=True, Triggerer
  • DagBag, dag.test(), @task_group

Chapter 4: SQL workloads

  • SQL with SQLExecuteQueryOperator
  • CronPartitionTimetable, PartitionedAssetTimetable
  • Data quality, CLI, productionize Airflow
Building Data Pipelines with Airflow

Start building today

Start building today

$$

Building Data Pipelines with Airflow

Happy building!

Building Data Pipelines with Airflow

Preparing Video For Download...