Single system data streaming

Streaming Concepts

Mike Metzger

Data Engineer

Intro to streaming

What is streaming?

  • Data doesn't stop until processed
    • Once initially processed, may have other data processing components
  • Is open-ended (no specific end event)
  • Is defined by the flow of data, not the content
Streaming Concepts

Logs

  • Store event information
  • Could be a simple text or binary file
  • Or a system to export information to multiple clients (ie, Apache Kafka)
  • Will store information until resources are exhausted / pruned
  • Purpose of the log depends on the application
210507-162356 - SUCCESS: Open vvlj45.txt
210507-162254 - ERROR: Open hjry57.txt failed
210507-161523 - SUCCESS: Open kbhn78.txt
210507-161235 - ERROR: Open ldge12.txt failed
210507-160127 - WARNING: keop98.txt exists
210507-155958 - SUCCESS: Open hqaz64.txt
210507-155439 - SUCCESS: Open neuf36.txt
210507-152335 - SUCCESS: Open mqpa91.txt
210507-144756 - ERROR: Open pqzi32.txt failed
210507-143541 - SUCCESS: Open urmn15.txt
210507-143152 - SUCCESS: Open fgty82.txt
210507-141732 - SUCCESS: Open mlwe96.txt
Streaming Concepts

System event log

  • Present on Windows, Mac, Linux
  • Processes and stores various system event information
  • Windows EventLog, Mac / Linux syslog

Components:

  • Listener: Accepts messages
  • Parser: Understands how to read messages
  • Logic: Decides what to do
  • Writer: Stores the messages for later
Streaming Concepts

Let's practice!

Streaming Concepts

Preparing Video For Download...