What is data engineering?

Introduction to Data Engineering

Vincent Vankrunkelsven

Data Engineer @ DataCamp

What to expect

  • Chapter 1
    • What is data engineering?
  • Chapter 2
    • Tools data engineers use
  • Chapter 3
    • Extract
    • Transform
    • Load
  • Chapter 4
    • Data engineering at DataCamp!
Introduction to Data Engineering

In comes the data engineer

 

  • Data is scattered
  • Not optimized for analyses
  • Legacy code is causing corrupt data

 

Data engineer to the rescue!

Female Engineer

Image of database schema to represent scattered data

Image representing a computer error

Introduction to Data Engineering

Data engineers: making your life easier

 

  • Gather data from different sources
  • Optimized database for analyses
  • Removed corrupt data

 

Data scientist's life got way easier!

 

Diagram representing the data ingestion process

Introduction to Data Engineering

Definition of the job

An engineer that develops, constructs, tests, and maintains architectures such as databases and large-scale processing systems

  • Processing large amounts of data
  • Use of clusters of machines
Introduction to Data Engineering

Data Engineer vs Data Scientist

Data Engineer

  • Develop scalable data architecture
  • Streamline data acquisition
  • Set up processes to bring together data
  • Clean corrupt data
  • Well versed in cloud technology

Data Scientist

  • Mining data for patterns
  • Statistical modeling
  • Predictive models using machine learning
  • Monitor business processes
  • Clean outliers in data
Introduction to Data Engineering

Let's practice!

Introduction to Data Engineering

Preparing Video For Download...