Created , Updated

Identify Kafka consumer performance issues by analyzing the Kafka consumer lag using Kafka’s CLI.

Step-by-Step Guide

1. Open a terminal which allows to use Kafka CLI commands

2. If desired, start a screen 

screen -S kafkacli
CODE

3. Watch the GetOffsetShell command

Issue the following command, inserting the relevant broker-list and topic name for your environment

watch -n 5 kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:10000 --topic ziris-smf
BASH
Output example:
Output format:

topic-name:partition-number:partition-offset

If partition-offset is increasing steadily, then this indicates that there is a steady workload on the specified topic.

If partition-offset is increasing rapidly, then this indicates a heavy workload on the specified topic.

4. Watch the kafka-cosumer-groups.sh command

Issue the following command, insert the relevant bootstrap-server and consumer group to observe lag results for the specified consumer group

watch -n 5 kafka-consumer-groups.sh --bootstrap-server localhost:10000 --group consumer-group1 --describe
CODE
Output example:

Monitor the LAG column for steadily or rapidly increasing consumer lag.

A steady increase may indicate that there is currently a higher workload than the consumer group can process.

A rapid increase may indicate that your consumer group is experiencing more severe issues, for example some or all consumers are currently inactive or unable to process any messages.

Possible workarounds

Restart all (inactive) consumers.

Increase the capacity to process high workloads and to ensure that messages are processed closer to real-time as possible by running additional consumers for this consumer group.