Airflow Sensors

Introduction to Apache Airflow in Python

Mike Metzger

Data Engineer

Sensors

  • An operator that waits for a certain condition to be true
    • Create of a file
    • Upload of a database record
    • Response from a web request
  • Can define how often to check for the condition to be true
  • Assigned to tasks

Illustration of an Airflow sensor waiting for a condition to become true

Introduction to Apache Airflow in Python

Sensor details

  • Derived from airflow.sdk.BaseSensorOperator
  • Sensor arguments:
  • mode - How to check for the condition
    • mode='poke' - The default, run repeatedly
    • mode='reschedule' - Give up task slot and try again later
  • poke_interval - How often to wait between checks
  • timeout - How long to wait before failing task
  • Also includes normal operator attributes
Introduction to Apache Airflow in Python

File sensor

  • Is part of the airflow.providers.standard.sensors library
  • Checks for the existence of a file at a certain location
  • Can also check if any files exist within a directory
from airflow.providers.standard.sensors.filesystem import FileSensor

file_sensor_task = FileSensor(task_id='file_sense',
                              filepath='salesdata.csv',
                              poke_interval=300,
                              timeout=3000
                              )

init_sales_cleanup() >> file_sensor_task >> generate_report()
Introduction to Apache Airflow in Python

Other sensors

  • Many others in airflow.providers.*.sensors
  • ExternalTaskSensor - wait for a task in another Dag to complete
  • HttpSensor - Request a web URL and check for content
  • SqlSensor - Runs a SQL query to check for content
Introduction to Apache Airflow in Python

When to use sensors?

  • Uncertain when a condition will be true
  • If failure not immediately desired
  • Add task repetition without loops

 

 

Illustration of an Airflow sensor waiting for a condition to become true

Introduction to Apache Airflow in Python

Let's practice!

Introduction to Apache Airflow in Python

Preparing Video For Download...