Queuing

Streaming Concepts

Mike Metzger

Data Engineer

What is queuing?

• Basically, a line
• Useful for processing in order
• First-in, first-out (FIFO)
• Sometimes referred to as a buffer
• Details vary a lot by implementation

¹ Photo by Joshua Tsu on Unsplash

Why queues?

• Queues allow tracking of processing order
• Can be processed by a single person / program or multiple
• Can be disconnected from the remainder of the processing pipeline
• Reasonably easy to scale vertically or horizontally
  • Vertical scaling by adding faster hardware
  • Horizontal scaling by adding more executors

Queue issues

• Bad data or processing errors
  • Customer pays with invalid credit card
• Data size variances
  • Supermarket fast lane with 100 items
• Sometimes difficult to know the length of the queue
  • First preview showing of a movie
• Scaling limits
  • Not enough space for more registers

Let's practice!

Streaming Concepts