Apache Kafka is a powerful platform for handling real-time data streams. One of the most common tasks when working with Kafka is consuming messages from topics. In this article, we’ll explore how to consume Kafka messages using the command-line interface (CLI).
Prerequisites
Before you get started, make sure you have the following:
- A running Kafka cluster.
- Kafka command-line tools installed on your system (part of the Kafka distribution).
Using the Kafka Console Consumer
The Kafka distribution comes with a handy command-line tool called kafka-console-consumer that allows you to consume messages from Kafka topics. Here’s how you can use it:
kafka-console-consumer.sh --bootstrap-server <kafka-broker>:<port> --topic <topic-name> [--from-beginning]
<kafka-broker>:<port>: Replace this with the address of your Kafka broker and the port it’s running on.<topic-name>: Specify the Kafka topic you want to consume messages from.--from-beginning (optional): Use this flag if you want to start consuming messages from the beginning of the topic.
Consuming Messages with Key and Value
Kafka messages consist of a key and a value. You can consume messages along with their keys using the --property print.key=true flag.
kafka-console-consumer.sh --bootstrap-server <kafka-broker>:<port> --topic <topic-name> --from-beginning --property print.key=true
Consuming Messages with Avro Serialization
If your Kafka topic uses Avro serialization for messages, you can consume and deserialize Avro messages using the
--from-beginning
--property value.deserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer properties.
kafka-console-consumer.sh --bootstrap-server <kafka-broker>:<port> --topic <topic-name> --from-beginning --property value.deserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer --property schema.registry.url=<schema-registry-url>
<schema-registry-url>: Replace this with the URL of your Avro schema registry.
Conclusion
In this article, we’ve covered the basics of consuming Kafka messages via the command-line interface (CLI). This approach is useful for quickly testing and debugging your Kafka topics without writing custom consumer code. The kafka-console-consumer tool is a powerful utility that every Kafka developer should be familiar with.
