Airflow executors

Introduction to Apache Airflow in Python

Mike Metzger

Data Engineer

What is an executor?

  • Executors run tasks
  • Different executors handle running the tasks differently
  • Example executors:
    • SequentialExecutor
    • LocalExecutor
    • KubernetesExecutor
Introduction to Apache Airflow in Python

SequentialExecutor

  • The default Airflow executor
  • Runs one task at a time
  • Useful for debugging
  • While functional, not really recommended for production
Introduction to Apache Airflow in Python

LocalExecutor

  • Runs on a single system
  • Treats tasks as processes
  • Parallelism defined by the user
  • Can utilize all resources of a given host system
Introduction to Apache Airflow in Python

KubernetesExecutor

  • Uses a Kubernetes as task manager
  • Multiple worker systems can be defined
  • Is significantly more difficult to setup & configure
  • Extremely powerful method for organizations with extensive workflows
Introduction to Apache Airflow in Python

Determine your executor

  • Via the airflow.cfg file
  • Look for the executor= line

cat airflow.cfg | grep "executor = "

Introduction to Apache Airflow in Python

Determine your executor #2

  • Via the first few lines of airflow info
  • executor | SequentialExecutor

airflow info

Introduction to Apache Airflow in Python

Let's practice!

Introduction to Apache Airflow in Python

Preparing Video For Download...