The data pipeline

Capire il Data Engineering

Hadrien Lacroix

Content Developer at DataCamp

If data is the new oil...

data is the new oil - economist cover

¹ The Economist, 2017-05-06, by David Parkins

oil well

piping from oil well

distilling

residue

heavy oil

diesel

kerosene

naphtha

gasoline

kerosene is delivered directly to airport

gasoline is delivered to gas storage facility

gasoline is delivered from gas storage facility to gas stations

naphtha undergoes chemical transformations

plastic is sent to the factory

Back to data engineering

Ingest
Process
Store
Need pipelines
Automate flow from one station to the next
Provide up-to-date, accurate, relevant data

data-engineer

mobile

computer

website

pipes from mobile app, desktop app and website

data rack

artists

albums

tracks

playlists

customers

employees

Artists database

sales employees

engineering employees

Support employees

United States sales employees

Belgium sales employees

France sales employees

check and clean tracks

write clean tracks to database

Oprah distributing pipelines

Data pipelines ensure an efficient flow of the data

Automate

Extracting
Transforming
Combining
Validating
Loading

Reduce

Human intervention
Errors
Time it takes data to flow

ETL and data pipelines

ETL

Popular framework for designing data pipelines
1) Extract data
2) Transform extracted data
3) Load transformed data to another database

Data pipelines

Move data from one system to another
May follow ETL
Data may not be transformed
Data may be directly loaded in applications

Summary

What a data pipeline is
What it does
Why it's important
How data pipelines are implemented at Spotflix
What ETL is and its nuances

Let's practice!

Capire il Data Engineering