Examples of the modern data architecture

Understanding Modern Data Architecture

Miller Trujillo

Senior Software Engineer

Lambda architecture

• Data processing architecture
• Enables processing of large-scale datasets
• Enables real-time data processing
• Combines batch and streaming
• 3 main layers:
  • Batch layer
  • Speed layer
  • Serving layer

Lambda architecture layers: Batch

• Master dataset
  • Immutable data
  • Append-only
• Pre-computed views
• Bank accounts
  • Master dataset: List of transactions
  • Views: Transactions applied to get the balance
• Perfect accuracy

Lambda architecture layers: Speed

• Batch jobs at a given schedule
• Gap between known data and reality
• Speed layer close that gap
• Not perfect accuracy

Lambda architecture layers: Serving

• Merge batch and real-time views
• Unified view of data

Complexity of Lambda architecture

• Two layers increase complexity
• Duplicated logic, code, algorithms
• Multiple stacks: Batch and streaming

Kappa architecture

Lambda vs. Kappa

Aspect	Lambda Architecture	Kappa Architecture
Batch Processing	Present	By processing the source of events again
Stream Processing	Present	Present
Complexity	Higher	Lower
Real-time Processing	Delayed by batch layer	Direct and streamlined
Use Cases	Complex analysis, historical data	Real-time analytics, real-time monitoring

Let's practice!

Understanding Modern Data Architecture