Airflow Variables

Introduction to Apache Airflow in Python

Mike Metzger

Data Engineer

What are variables?

  • Useful for configuration values
  • File paths, API endpoints, environments, etc
  • Can be queried from tasks, set via code, Airflow UI, or via file
  • Exist outside of Dag code
Introduction to Apache Airflow in Python

Accessing variables in Airflow UI

  • Via the Admin: Variables menu

Airflow UI with the Admin menu open showing the Variables option

Introduction to Apache Airflow in Python

Viewing variables in Airflow UI

Airflow Variables page with an empty list and no variables defined

Introduction to Apache Airflow in Python

Adding a variable in Airflow

Airflow Variables page with the Add Variable button highlighted

Introduction to Apache Airflow in Python

Variable editor

Airflow variable editor form with key, value, and description fields

Introduction to Apache Airflow in Python

Variables present in the UI

Airflow Variables page showing one defined variable in the list

Introduction to Apache Airflow in Python

Reading a variable in Python

  • Uses the airflow.sdk.Variable library
  • Read via Variable.get("variable_name")
    # Retrieve a plain string value
my_region = Variable.get("region")
  • In this case, my_region would be us-eastern
  • Provide a fallback if the variable doesn't exist
    # Provide a fallback if the variable doesn't exist
env = Variable.get("current_environment", default=None)
  • As we didn't define a current_environment, env is None
Introduction to Apache Airflow in Python

Accessing variables in Jinja templates

  • Accessible via the {{ var.value.variable_name }}
    filewatcher = FileSensor(
task_id="wait_for_files",
filepath="/data/{{ var.value.region }}/input.csv",
...
  • In our case, the FileSensor would look for the file /data/us-eastern/input.csv
Introduction to Apache Airflow in Python

Accessing variables through CLI

  • Variables can be read, updated, deleted or listed via command-line
  • Read - airflow variables get <variable_name>
  • Write - airflow variables set <variable_name> <value>
  • Delete - airflow variables delete <variable_name>
  • List - airflow variables list <variable_name>
Introduction to Apache Airflow in Python

JSON variables

  • Variable values can be JSON strings
    • Variable.get("variable_name", deserialize_json=True)
    • Returns a Python dictionary
  • Using Jinja
    • {{ var.json.variable_name }}
Introduction to Apache Airflow in Python

Setting variables

  • Possible to set variables as well
  • Using Variable.set("variable_name", variable_value)
  • Can run JSON as well with the serialize_json=True option
  • Use sparingly as this can affect other Dags accessing variables in unforeseen ways
Introduction to Apache Airflow in Python

Let's practice!

Introduction to Apache Airflow in Python

Preparing Video For Download...