Intro to batch processing

Streaming Concepts

Mike Metzger

Data Engineer

What is batch processing?

• Processing data in groups
• Runs from start of process to finish
  • No data added in between
• Typically run as result of
  • an interval
  • starting event
• Processed in a certain size (batch size)
• An instance of a batch process is often referred to as a job

Common batch processing scenarios

• Reading files or parts of files (text, mp3, etc)
• Sending / receiving email
• Printing

Why batch?

• Simple
• Generally consistent
• Multiple ways to improve performance

Let's practice!

Streaming Concepts