Airflow executors

Introdução ao Apache Airflow em Python

Mike Metzger

Data Engineer

What is an executor?

  • Executors run tasks
  • Different executors handle running the tasks differently
  • Example executors:
    • SequentialExecutor
    • LocalExecutor
    • KubernetesExecutor
Introdução ao Apache Airflow em Python

SequentialExecutor

  • The default Airflow executor
  • Runs one task at a time
  • Useful for debugging
  • While functional, not really recommended for production
Introdução ao Apache Airflow em Python

LocalExecutor

  • Runs on a single system
  • Treats tasks as processes
  • Parallelism defined by the user
  • Can utilize all resources of a given host system
Introdução ao Apache Airflow em Python

KubernetesExecutor

  • Uses a Kubernetes as task manager
  • Multiple worker systems can be defined
  • Is significantly more difficult to setup & configure
  • Extremely powerful method for organizations with extensive workflows
Introdução ao Apache Airflow em Python

Determine your executor

  • Via the airflow.cfg file
  • Look for the executor= line

cat airflow.cfg | grep "executor = "

Introdução ao Apache Airflow em Python

Determine your executor #2

  • Via the first few lines of airflow info
  • executor | SequentialExecutor

airflow info

Introdução ao Apache Airflow em Python

Let's practice!

Introdução ao Apache Airflow em Python

Preparing Video For Download...