Kafka topics

Introduction to Apache Kafka

Mike Metzger

Data Engineering Consultant

What is a topic?

Topics are logical groupings of events
- Similar to a table in a relational database
- Event log
- Messages are immutable
- Messages in a topic can be read or written, but not modified
- Messages can be removed based on age

Kafka topic illustration.png

Creating topics

Multiple ways to create topics

We'll use the bin/kafka-topics.sh script

bin/kafka-topics.sh 
    --bootstrap-server <server> 
    --topic <topicname>
    --create

Example

$ bin/kafka-topics.sh --bootstrap-server localhost:9092 \
--topic orders --create

Created topic orders

Other variations

Optional arguments for bin/kafka-topics.sh --create
- --replication-factor <x>
  - Define the replication factor of the topic
- --partitions <x>
  - Specify the number of partitions manually

Example

$ bin\kafka-topics.sh --bootstrap-server localhost:9092 \
--topic orders --create --replication-factor 3 \
--partitions 3

Created topic orders

--describe

bin/kafka-topics.sh --bootstrap-server <server> --topic <topicname> --describe
- Get details about the topic configuration

Example

$ bin/kafka-topics.sh --bootstrap-server localhost:9092 \
   --topic orders --describe

Topic: orders-new TopicId: <topicid> PartitionCount: 3 ReplicationFactor: 1    
  Configs:
    Topic: orders-new    Partition: 0    Leader: 0    Replicas: 0    Isr: 0
    Topic: orders-new    Partition: 1    Leader: 0    Replicas: 0    Isr: 0
    Topic: orders-new    Partition: 2    Leader: 0    Replicas: 0    Isr: 0

Removing topics

bin/kafka-topics.sh --bootstrap-server <server> --topic <topicname> --delete
- Removes a topic from Kafka server
- Removing the topic also removes all messages in the topic

Example

$ bin/kafka-topics.sh --bootstrap-server localhost:9092
  --topic orders --delete

Let's practice!

Introduction to Apache Kafka