Introduction to Apache Airflow in Python
Mike Metzger
Data Engineer
Tasks are:
example_task = BashOperator(task_id='bash_example',
bash_command='echo "Example!"')
$$
Upstream means before
Downstream means after
# Define the tasks task1 = BashOperator(task_id='first_task', bash_command='echo 1' )
task2 = BashOperator(task_id='second_task', bash_command='echo 2' )
# Set first_task to run before second_task task1 >> task2 # or task2 << task1
Chained dependencies:
task1 >> task2 >> task3 >> task4
Mixed dependencies:
task1 >> task2 << task3
or:
task1 >> task2
task3 >> task2
Introduction to Apache Airflow in Python