Kafka producers

Introduction to Apache Kafka

Mike Metzger

Data Engineering Consultant

What is a producer?

  • Writes messages to Kafka topics
    • Sometimes called publisher
  • Messages sent via producers are stored on Kafka for later use
  • Can have many producers
  • Producers can write to one or multiple topics
Introduction to Apache Kafka

Types of producers

  • Many different kinds of producers
    • Command-line
      • kafka-console-producer.sh
    • Python
    • Java
    • Many other languages
    • Kafka Connect
Introduction to Apache Kafka

kafka-console-producer.sh

  • Found in bin/ folder, with other Kafka tools
    • bin/kafka-console-producer.sh
  • Several command-line options, some required, others not
  • Required options:
    • --bootstrap-server - Specifies which server to use
      • --bootstrap-server localhost:9092
    • --topic
      • --topic phishing-sites
  • By default, opens a connection where messages can be typed
    • Use Ctrl-C to exit
  • Can also use | symbol to pass messages
Introduction to Apache Kafka

Example

  • Interactive example
bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic testing

>This is the first message
>This is the second message
>This is the third message
<ctrl+c to exit>
  • Non-interactive
echo "This is the first message" |
bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic testing
Introduction to Apache Kafka

Let's practice!

Introduction to Apache Kafka

Preparing Video For Download...