Orchestration

Understanding Modern Data Architecture

Miller Trujillo

Senior Software Engineer

What is orchestration?

High level workflow to process 5 tables

  • Coordinate multiple jobs
  • Automated configuration and coordination of complex workflows.

Airflow sample DAG

  • Frees up human resources
1 https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html
Understanding Modern Data Architecture

Orchestration vs scheduling

Scheduling

  • Execute tasks at specified intervals/times
  • Starter of orchestrated workflows

Scheduling example

Orchestration

Automate and coordinate complex workflows

Orchestration example

Understanding Modern Data Architecture

Apache Airflow

Apache Airflow logo

  • Coordinate complex workflows with Python
Understanding Modern Data Architecture

Core concepts of orchestration

  • Tasks:
    • Basic unit of execution
  • Dependencies:
    • Determine task sequence
  • Directed Acyclic Graph (DAG):
    • Workflow of tasks and dependencies

Airflow DAG sample

Understanding Modern Data Architecture

Core concepts of orchestration

  • Operators:
    • Determine nature of task
    • BashOperator
    • PythonOperator
  • Sensors:
    • Wait for specific conditions
  • Scheduler
    • Automates triggering of tasks.
Understanding Modern Data Architecture

Let's practice!

Understanding Modern Data Architecture

Preparing Video For Download...