Introduction to Apache Kafka

Introduction to Apache Kafka

Mike Metzger

Data Engineering Consultant

About me

  • 25+ year IT career, 18+ in data
  • Tools:
    • Apache Spark
    • Apache Airflow
    • Apache Kafka

Apache_Spark_logo.svg.png

1600px-AirflowLogo.png

Apache Kafka Logo

Introduction to Apache Kafka

What is Apache Kafka?

  • Open-source
  • Distributed
  • Event streaming
  • Designed to handle large quantities of data
  • Scalable

Apache Kafka Logo

Introduction to Apache Kafka

Event streaming in Kafka

  • Obtain source information
  • Stored reliably
  • Distribute data to clients
Introduction to Apache Kafka

Common uses

  • Ecommerce
  • Order tracking
  • Ride-share / Food delivery
  • Sensor data
  • Cybersecurity

Computer image

1 Photo by Carlos Muza on Unsplash
Introduction to Apache Kafka

Kafka components

  • User facing components

Kafka base image

Introduction to Apache Kafka

Kafka components

  • User facing components
    • Topics

Kafka with Topics

Introduction to Apache Kafka

Kafka components

  • User facing components
    • Topics
    • Producers (writers)

Kafka with Producers

Introduction to Apache Kafka

Kafka components

  • User facing components
    • Topics
    • Producers (writers)
    • Consumers (readers)

Kafka Consumer & Producer

Introduction to Apache Kafka

Kafka components

  • User facing components
    • Topics
    • Producers (writers)
    • Consumers (readers)

Kafka Consumer & Producer showing writes to topic

Introduction to Apache Kafka

Kafka components

  • User facing components
    • Topics
    • Producers (writers)
    • Consumers (readers)

Kafka Consumer & Producer - showing reads and writes

Introduction to Apache Kafka

Let's practice!

Introduction to Apache Kafka

Preparing Video For Download...