Workflow scheduling frameworks

Introduction to Data Engineering

Vincent Vankrunkelsven

Data Engineer, DataCamp

An example pipeline

Example simple pipeline that extracts from csv using Spark

How to schedule?

Manually
cron scheduling tool
What about dependencies?

DAGs

Directed Acyclic Graph

Set of nodes
Directed edges
No cycles

Example DAG

The tools for the job

Linux's cron
Prefect and Dagster
Apache Airflow

Logo of Apache Airflow

Created at Airbnb
DAGs
Python

Airflow: an example DAG

Example Airflow DAG

Airflow: an example in code

@dag(dag_id="example_dag",
     start_date=datetime(2024, 1, 1),
     schedule="0 * * * *")
def example_dag():

    @task
    def start_cluster(): ...
    @task
    def ingest_customer_data(): ...
    @task
    def ingest_product_data(): ...
    @task
    def enrich_customer_data(): ...

Airflow: an example in code

@dag(dag_id="example_dag", ...)
def example_dag():
    ...
    # Set up dependency flow
    cluster = start_cluster()
    customers = ingest_customer_data()
    products = ingest_product_data()
    cluster >> [customers, products]
    [customers, products] >> enrich_customer_data()
# Run the DAG
example_dag()

Let's practice!

Introduction to Data Engineering