Data Processing

Understanding Modern Data Architecture

Miller Trujillo

Senior Software Engineer

What is data processing?

• Exploration
• Data quality: Checks and transformations
• Analysis
• Aggregations
• Transformations

Batch processing

• Batch and streaming
• Fixed amount of data

Streaming processing

Fixed time window

Sliding time window

¹ https://beam.apache.org/documentation/programming-guide/#windowing

Streaming processing concepts

• When data was generated
• When data arrived
• Watermarks
• Late data
• Trigger new processing

Processing technologies

Use case	Solution	Cloud solution
Batch/streaming, big data, cluster	Apache Spark, Flink, Beam	AWS EMR, AWS Glue, Google Dataproc, Google Dataflow
Batch/streaming, big data, serverless (servers are fully managed by the provider)	Apache Spark, Beam	AWS Glue, Google Dataflow
Individual events, simple processing, 24/7 support without servers running	General programming languages: Python, Javascript, C#, Java, Go	AWS Lambda, Google Cloud Functions

Let's practice!

Understanding Modern Data Architecture