Introduction to Apache Airflow in Python
Mike Metzger
Data Engineer
What is a sensor?
airflow.sensors.base_sensor_operator
mode
- How to check for the conditionmode='poke'
- The default, run repeatedlymode='reschedule'
- Give up task slot and try again laterpoke_interval
- How often to wait between checkstimeout
- How long to wait before failing taskairflow.sensors
libraryfrom airflow.sensors.filesystem import FileSensor
file_sensor_task = FileSensor(task_id='file_sense',
filepath='salesdata.csv',
poke_interval=300,
dag=sales_report_dag)
init_sales_cleanup >> file_sensor_task >> generate_report
ExternalTaskSensor
- wait for a task in another DAG to completeHttpSensor
- Request a web URL and check for contentSqlSensor
- Runs a SQL query to check for contentairflow.sensors
and airflow.providers.*.sensors
Use a sensor...
Introduction to Apache Airflow in Python