SLAs and reporting in Airflow

Introduction to Apache Airflow in Python

Mike Metzger

Data Engineer

SLAs

What is an SLA?

An SLA stands for Service Level Agreement
Within Airflow, the amount of time a task or a DAG should require to run
An SLA Miss is any time the task / DAG does not meet the expected timing
If an SLA is missed, an email is sent out and a log is stored.
You can view SLA misses in the web UI.

SLA Misses

Found under Browse: SLA Misses

SLA Misses view

Defining SLAs

Using the 'sla' argument on the task

task1 = BashOperator(task_id='sla_task', 
                   bash_command='runcode.sh',
                   sla=timedelta(seconds=30),
                   dag=dag)

On the default_args dictionary

default_args={
 'sla': timedelta(minutes=20),
 'start_date': datetime(2023,2,20)
}
dag = DAG('sla_dag', default_args=default_args)

timedelta object

In the datetime library
Accessed via from datetime import timedelta

Takes arguments of days, seconds, minutes, hours, and weeks

timedelta(seconds=30)
timedelta(weeks=2)
timedelta(days=4, hours=10, minutes=20, seconds=30)

General reporting

Options for success / failure / error
Keys in the default_args dictionary

default_args={
  'email': ['[email protected]'],
  'email_on_failure': True,
  'email_on_retry': False,
  'email_on_success': True,
  ...
}

Within DAGs from the EmailOperator

Let's practice!

Introduction to Apache Airflow in Python