Introduction to Apache Kafka

Introduzione ad Apache Kafka

Mike Metzger

Data Engineering Consultant

About me

  • 25+ year IT career, 18+ in data
  • Tools:
    • Apache Spark
    • Apache Airflow
    • Apache Kafka

Apache_Spark_logo.svg.png

1600px-AirflowLogo.png

Apache Kafka Logo

Introduzione ad Apache Kafka

What is Apache Kafka?

  • Open-source
  • Distributed
  • Event streaming
  • Designed to handle large quantities of data
  • Scalable

Apache Kafka Logo

Introduzione ad Apache Kafka

Event streaming in Kafka

  • Obtain source information
  • Stored reliably
  • Distribute data to clients
Introduzione ad Apache Kafka

Common uses

  • Ecommerce
  • Order tracking
  • Ride-share / Food delivery
  • Sensor data
  • Cybersecurity

Computer image

1 Photo by Carlos Muza on Unsplash
Introduzione ad Apache Kafka

Kafka components

  • User facing components

Kafka base image

Introduzione ad Apache Kafka

Kafka components

  • User facing components
    • Topics

Kafka with Topics

Introduzione ad Apache Kafka

Kafka components

  • User facing components
    • Topics
    • Producers (writers)

Kafka with Producers

Introduzione ad Apache Kafka

Kafka components

  • User facing components
    • Topics
    • Producers (writers)
    • Consumers (readers)

Kafka Consumer & Producer

Introduzione ad Apache Kafka

Kafka components

  • User facing components
    • Topics
    • Producers (writers)
    • Consumers (readers)

Kafka Consumer & Producer showing writes to topic

Introduzione ad Apache Kafka

Kafka components

  • User facing components
    • Topics
    • Producers (writers)
    • Consumers (readers)

Kafka Consumer & Producer - showing reads and writes

Introduzione ad Apache Kafka

Let's practice!

Introduzione ad Apache Kafka

Preparing Video For Download...