Congratulations

Introduction to Data Engineering

Vincent Vankrunkelsven

Data Engineer @ DataCamp

Introduction to data engineering

 

• Identify the tasks of a data engineer
• What kind of tools they use
• Cloud service providers

Data engineering toolbox

 

• Databases
• Parallel computing & frameworks (Spark)
• Workflow scheduling with Airflow

Extract, Load and Transform (ETL)

 

• Extract: get data from several sources
• Transform: perform transformations using parallel computing
• Load: load data into target database

Case study: DataCamp

 

• Fetch data from multiple sources
• Transform to form recommendations
• Load into target database

Good job!

Introduction to Data Engineering