Data integration

Database Design

Lis Sulmont

Curriculum Manager

What is data integration

 

 

 

Data Integration combines data from different sources, formats, technologies to provide users with a translated and unified view of that data.

Database Design

Business case examples

  • 360-degree customer view
  • Acquisition
  • Legacy systems
Database Design

Unified data model

Database icon representing a unified data model

Database Design

Data sources

Three data sources with no connection to the unified data model

Database Design

Data sources format

Three data sources with no connections to the unified data model and with different formats

Database Design

Unified data model format

Three data sources with different formats unconnected to the unified data model which is a Redshift data warehouse

Database Design

Example: DataCamp

Three data sources with different formats and from different departments unconnected to the unified data model which is a Redshift data warehouse

Database Design

Update cadence - sales

The three data sources updating daily

Database Design

Update cadence - air traffic

The three data sources updating in real-time

Database Design

Different update cadences

The three data sources updating at different times

Database Design

So simple?

The three data sources connect to the unified data model

Database Design

Not really

The three data sources updating daily with a big X over it

Database Design

Transformations

The three data sources being transformed to fit the unified data model

Database Design

Transformation - tools

Processing the three data sources via ETL to the unified data model

Database Design

Choosing a data integration tool

  • Flexible
  • Reliable
  • Scalable
Database Design

Automated testing and proactive alerts

ETL diagram with testing icons at each phase

Database Design

Security

ETL diagram showing portions of data that should be restricted from its source to destination

Database Design

Security - credit card anonymization

ETL diagram with anonymization of credit card data

Database Design

Data governance - lineage

ETL diagram with a line representing the flow of data from its source to destination

Database Design

Let's practice!

Database Design

Preparing Video For Download...