Data serving

Understanding Modern Data Architecture

Miller Trujillo

Senior Software Engineer

Serving layer

Generic Big data architecture including data sources, ingestion, storage, processing, orchestration, governance, serving, and analytics storage and reporting

  • Where to store processed data
  • Protocols to consume it
Understanding Modern Data Architecture

What data do we have?

  • What data do we have?
    • Is data structured?
    • Is our data a time series?
  • How data will be consumed?

Data repository

  • Data Warehouse
    • Automated Reporting/Dashboarding
    • BI/Data Analytics
    • Direct Queries
  • Blob storage
  • Time series database
  • NoSQL database
Understanding Modern Data Architecture

How will data be consumed?

  • Data warehouse
    • Automated reporting/dashboarding
    • BI/Data analytics
    • Direct queries
  • Machine learning models
  • Applications
    • Summarized information
    • Individual records
  • Not one size fits all!

Python & R

API/Compute

Understanding Modern Data Architecture

Serving your data depending on your use case

System What type of data Consuming use case Tools
Data warehouse Structured data, can handle semi-structured data Analytical queries, BI/Analytics, reporting/dashboards BigQuery, Redshift, Snowflake
Blob storage All type of data ML models with video/imaging processing, archiving S3, Cloud Storage, Azure Storage
NoSQL database Semi-structured data, time series API exposure, high demand BigTable, DataStore, DynamoDB, CosmosDB
RDMBS Structured data, can handle semi-structured data Individual records consumption, API exposure, lower demand Postgres, MySQL, Oracle
Understanding Modern Data Architecture

Serving vs. Consuming

Data Serving vs. Consuming

Understanding Modern Data Architecture

Let's practice!

Understanding Modern Data Architecture

Preparing Video For Download...