Vertically scaling streaming systems

Streaming Concepts

Mike Metzger

Data Engineer

Why scale?

• Process the same data in less time
• Process more data in the same time
• Deliver data more quickly (reduce latency)
• Meet guarantees (SLAs)

Vertical scaling

• Improve the capabilities of a single system
• Faster / better components
  • CPU, RAM, Disk, Network
• All can affect streaming performance

Faster CPU / GPU performance

• Faster execution
• Better execution
  • New / improved instruction sets
• GPU processing
  • Machine learning
  • Deep learning
  • Image processing
  • Matrix operations

How does this affect streaming?

• Streaming processes don't stop until complete
• Different items can be in different parts of the pipeline, but total processing capacity is limited by the system performance
• Certain components have a greater effect than others, depending on workload
• Benchmark / test!

Let's practice!

Streaming Concepts