Branching

Introduction to Apache Airflow in Python

Mike Metzger

Data Engineer

Branching

Branching in Airflow:

  • Provides conditional logic
  • Using BranchPythonOperator
  • from airflow.operators.python import BranchPythonOperator
  • Takes a python_callable to return the next task id (or list of ids) to follow
Introduction to Apache Airflow in Python

Branching example

def branch_test(**kwargs):
  if int(kwargs['ds_nodash']) % 2 == 0:
    return 'even_day_task'
  else:
    return 'odd_day_task'
Introduction to Apache Airflow in Python

Branching example

def branch_test(**kwargs):
  if int(kwargs['ds_nodash']) % 2 == 0:
    return 'even_day_task'
  else:
    return 'odd_day_task'

branch_task = BranchPythonOperator(task_id='branch_task',dag=dag, provide_context=True, python_callable=branch_test)
start_task >> branch_task >> even_day_task >> even_day_task2
branch_task >> odd_day_task >> odd_day_task2
Introduction to Apache Airflow in Python

Branching graph view

Branching DAG graph view

Introduction to Apache Airflow in Python

Branching even days

Branching graph view - Even days

Introduction to Apache Airflow in Python

Branching odd days

Branching graph view - odd days

Introduction to Apache Airflow in Python

Let's practice!

Introduction to Apache Airflow in Python

Preparing Video For Download...