Skip to main content
Skip table of contents

Kafka CLI - How to identify Kafka consumer performance issues

Created , Updated


Identify Kafka consumer performance issues by analyzing the Kafka consumer lag using Kafka’s CLI.

Step-by-Step Guide


1. Open a terminal which allows to use Kafka CLI commands

2. If desired, start a screen

CODE
screen -S kafkacli

3. Watch the GetOffsetShell command

Issue the following command, inserting the relevant broker-list and topic name for your environment

BASH
watch -n 5 kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:10000 --topic ziris-smf
Output example:
Output format:

topic-name:partition-number:partition-offset

If partition-offset is increasing steadily, then this indicates that there is a steady workload on the specified topic.

If partition-offset is increasing rapidly, then this indicates a heavy workload on the specified topic.

4. Watch the kafka-cosumer-groups.sh command

Issue the following command, insert the relevant bootstrap-server and consumer group to observe lag results for the specified consumer group

CODE
watch -n 5 kafka-consumer-groups.sh --bootstrap-server localhost:10000 --group consumer-group1 --describe
Output example:

Monitor the LAG column for steadily or rapidly increasing consumer lag.

A steady increase may indicate that there is currently a higher workload than the consumer group can process.

A rapid increase may indicate that your consumer group is experiencing more severe issues, for example some or all consumers are currently inactive or unable to process any messages.

Possible workarounds

Restart all (inactive) consumers.

Increase the capacity to process high workloads and to ensure that messages are processed closer to real-time as possible by running additional consumers for this consumer group.

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.