Deferrable operators

Building Data Pipelines with Airflow

Volker Janz

Senior Developer Advocate at Astronomer

The sensor problem

Sensors occupying worker slots

Poke mode: sensor holds the worker slot the entire time
Long-running sensors starve other tasks

Reschedule mode

Poke mode (default)

from airflow.providers.standard.sensors.filesystem \
    import FileSensor

wait = FileSensor(
    task_id="wait",
    filepath="/data/report.csv",
    mode="poke",
    poke_interval=30,
)

Slot occupied entire time

Reschedule mode

from airflow.providers.standard.sensors.filesystem \
    import FileSensor

wait = FileSensor(
    task_id="wait",
    filepath="/data/report.csv",
    mode="reschedule",
    poke_interval=300,
)

Slot freed between checks

Deferrable mode

Deferred sensor handled by the Triggerer

Task hands off to the Triggerer process
Triggerer uses async I/O for hundreds of concurrent waits
Zero worker slots used during the wait

Enabling deferrable

from airflow.providers.standard.sensors.filesystem import FileSensor

wait_for_data = FileSensor(
    task_id="wait_for_data",
    filepath="/data/incoming/report.csv",

    deferrable=True,

    poke_interval=30,
    timeout=3600,
)

poke_interval and timeout still apply

When to use each mode

Mode	Best for	Worker slot
poke	Short waits (< 1 min)	Held entire time
reschedule	Moderate waits, no Triggerer	Freed between checks
deferrable	Long waits, many sensors	Never used

If the Triggerer is running, deferrable is almost always the right default

Global deferrable setting

AIRFLOW__OPERATORS__DEFAULT_DEFERRABLE=True

All compatible operators use deferrable mode automatically
No changes to Dag code needed
Override per-operator with deferrable=False if needed

Let's practice!

Building Data Pipelines with Airflow