Kafka CLI - How to identify Kafka consumer performance issues

Created 25 May 2021, Updated 25 May 2021

Identify Kafka consumer performance issues by analyzing the Kafka consumer lag using Kafka’s CLI.

Step-by-Step Guide

1. Open a terminal which allows to use Kafka CLI commands

2. If desired, start a screen

screen -S kafkacli

3. Watch the GetOffsetShell command

Issue the following command, inserting the relevant broker-list and topic name for your environment

watch -n 5 kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:10000 --topic ziris-smf

Output example:

Output format:

topic-name:partition-number:partition-offset

If partition-offset is increasing steadily, then this indicates that there is a steady workload on the specified topic.

If partition-offset is increasing rapidly, then this indicates a heavy workload on the specified topic.

4. Watch the kafka-cosumer-groups.sh command

Issue the following command, insert the relevant bootstrap-server and consumer group to observe lag results for the specified consumer group

watch -n 5 kafka-consumer-groups.sh --bootstrap-server localhost:10000 --group consumer-group1 --describe

Output example:

Monitor the LAG column for steadily or rapidly increasing consumer lag.

A steady increase may indicate that there is currently a higher workload than the consumer group can process.

A rapid increase may indicate that your consumer group is experiencing more severe issues, for example some or all consumers are currently inactive or unable to process any messages.

Possible workarounds

Restart all (inactive) consumers.

Increase the capacity to process high workloads and to ensure that messages are processed closer to real-time as possible by running additional consumers for this consumer group.