Debugging and troubleshooting in Airflow

Introduction to Apache Airflow in Python

Mike Metzger

Data Engineer

Common issues

Dag won't run on schedule
Dag won't load
Syntax errors

Illustration representing common Airflow troubleshooting issues

Dag won't run on schedule

At least one schedule period hasn't passed
- Modify the attributes to meet your requirements
Not enough resources to run
- Add system resources
- Add more systems
- Change Dag scheduling

Dag won't load

Dag not in web UI
Dag not in airflow dags list

Possible solutions

Verify Dag file is in correct folder (dag files aren't parsed outside Dags folder)
Determine the Dags folder via airflow info

Terminal output of airflow info highlighting the dags_folder path

Syntax errors

The most common reason a Dag file won't appear
Sometimes difficult to find errors in Dag
Three quick methods:
- Check the Airflow UI for errors popup
- Run airflow dags list-import-errors
- Run python3 <dagfile.py>

Airflow UI import errors

Airflow welcome page showing a Dag import error alert

Airflow import error detail showing the Dag traceback message

airflow dags list-import-errors

Terminal output of airflow dags list-import-errors showing an error

airflow dags reserialize

Dag changes may not appear immediately
By default, Airflow will check for Dag updates every 5 minutes
Can force a re-check via airflow dags reserialize

Terminal running airflow dags reserialize to force a Dag refresh

Running the Python interpreter

python3 dagfile.py:

With errors

Terminal showing a Python syntax error from running a Dag file with python3

Without errors

Terminal returning to the prompt after running a Dag file with python3 and no errors

Let's practice!

Introduction to Apache Airflow in Python

Preparing Video For Download...