Data serving

Understanding Modern Data Architecture

Miller Trujillo

Senior Software Engineer

Serving layer

Generic Big data architecture including data sources, ingestion, storage, processing, orchestration, governance, serving, and analytics storage and reporting

Where to store processed data
Protocols to consume it

What data do we have?

What data do we have?
- Is data structured?
- Is our data a time series?
How data will be consumed?

Data repository

Data Warehouse
- Automated Reporting/Dashboarding
- BI/Data Analytics
- Direct Queries
Blob storage
Time series database
NoSQL database

How will data be consumed?

Data warehouse
- Automated reporting/dashboarding
- BI/Data analytics
- Direct queries
Machine learning models
Applications
- Summarized information
- Individual records
Not one size fits all!

Python & R

API/Compute

Serving your data depending on your use case

System	What type of data	Consuming use case	Tools
Data warehouse	Structured data, can handle semi-structured data	Analytical queries, BI/Analytics, reporting/dashboards	BigQuery, Redshift, Snowflake
Blob storage	All type of data	ML models with video/imaging processing, archiving	S3, Cloud Storage, Azure Storage
NoSQL database	Semi-structured data, time series	API exposure, high demand	BigTable, DataStore, DynamoDB, CosmosDB
RDMBS	Structured data, can handle semi-structured data	Individual records consumption, API exposure, lower demand	Postgres, MySQL, Oracle

Serving vs. Consuming

Data Serving vs. Consuming

Let's practice!

Understanding Modern Data Architecture