Kafka consumers

Introduction to Apache Kafka

Mike Metzger

Data Engineering Consultant

What is a consumer?

  • Reads messages from Kafka topics
    • Sometimes called subscribers
  • Kafka provides messages stored in topics to consumers
  • Can have many consumers
  • Consumers can read from one or multiple topics
Introduction to Apache Kafka

Types of consumers

  • Many different kinds of consumers
    • Command-line
      • kafka-console-consumer.sh
    • Python
    • Java
    • Many other languages
    • Kafka Connect
Introduction to Apache Kafka

kafka-console-consumer.sh

  • Found in bin/ folder, with other Kafka tools
    • bin/kafka-console-consumer.sh
  • Several options, some required
  • Required options same as kafka-console-producer:
    • --topic
    • --bootstrap-server
  • By default, opens a connection where messages appear (real-time) and maintain position.
  • Use Ctrl-C to exit
Introduction to Apache Kafka

Optional arguments

  • --from-beginning
    • Tells the Kafka system to send all messages in topic, even ones that may have been already read.
  • --max-messages <number>
    • Used to read up to a maximum number of messages before exiting
Introduction to Apache Kafka

Interactive example

  • Interactive example
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
    --topic testing --from-beginning

This is the first message
This is the second message
This is the third message
<ctrl+c to exit>
Introduction to Apache Kafka

Non-interactive example

  • Non-interactive
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 /
  --topic testing --from-beginning --max-messages 2

This is the first message
This is the second message
<automatically exits>
Introduction to Apache Kafka

Let's practice!

Introduction to Apache Kafka

Preparing Video For Download...