Working with templates

Introduction to Apache Airflow in Python

Mike Metzger

Data Engineer

What are templates?

Templates:

  • Allow substituting information during a DAG run
  • Provide added flexibility when defining tasks
  • Are created using the Jinja templating language
Introduction to Apache Airflow in Python

Non-Templated BashOperator example

Create a task to echo a list of files:

t1 = BashOperator(
       task_id='first_task',
       bash_command='echo "Reading file1.txt"',
       dag=dag)

t2 = BashOperator( task_id='second_task', bash_command='echo "Reading file2.txt"', dag=dag)
Introduction to Apache Airflow in Python

Templated BashOperator example

templated_command="""
  echo "Reading {{ params.filename }}"
"""

t1 = BashOperator(task_id='template_task', bash_command=templated_command, params={'filename': 'file1.txt'}, dag=example_dag)

Output:

Reading file1.txt
Introduction to Apache Airflow in Python

Templated BashOperator example (continued)

templated_command="""
  echo "Reading {{ params.filename }}"
"""

t1 = BashOperator(task_id='template_task', bash_command=templated_command, params={'filename': 'file1.txt'}, dag=example_dag)
t2 = BashOperator(task_id='template_task', bash_command=templated_command, params={'filename': 'file2.txt'}, dag=example_dag)
Introduction to Apache Airflow in Python

Let's practice!

Introduction to Apache Airflow in Python

Preparing Video For Download...