Kafka record extractor - User Guide
Created , Updated
The Kafka record extractor deploys a kafkacat container on Docker. It extracts SMF records, filtered by a configured time interval, from an Apache Kafka cluster. In some troubleshooting cases, support teams may request that an extract of SMF records. The following instructions help users to install, configure and run the Kafka record extractor.
Step 1: Prerequisites
A Linux distribution with Docker installed is required.
Step 2: Installation
New users should contact technical support to receive their access credentials.
kafka-record-extractor.tar archive to a desired location from our JFrog repository.
Extract the archive:
tar -xvf kafka-record-extractor.tar
Step 3: Configuration
Customize the properties in the
settings.config configuration file:
The character # can be used at the beginning of a line to comment out that line.
Colons are used as delimiters to distinguish between keys and values, for example:
Colons may not be used as part of a key nor a value.
A value can be represented as a bash command by enclosing them with
$(<command>), for example:
KAFKA_HOST_IP: $(ip -4 addr show docker0 | grep -Po 'inet \\K[\\d.]+')CODE
Any leading and/or trailing spaces in keys and values are trimmed during processing
date system command formats the string in START_TIMEDATE and END_TIMEDATE into a corresponding timestamp. Click here for more information about the supported date string formats.
The Kafka cluster IP address or hostname
The active port listening for connections to the Kafka cluster.
The existing topic that contains the SMF records.
2021-04-12 20:52:00 UTC
Earliest time a record arrived in the cluster which should be extracted. Forms a time interval with END_TIMEDATE.
2021-04-12 21:52:00 UTC
Latest time a record arrived in the cluster which should be extracted. Forms a time interval with START_TIMEDATE.
The maximum number of records to be extracted.
Flag determines whether extracted records must be archived into gzipped tarballs.
Only processed if ARCHIVE_RECORDS is true.
The maximum number of records per archive.
When set to 0, all extracted records wil be archived into a single gzipped tarball.
Step 4: Run
The properties must be configured correctly before running the Kafka record extractor
Run the bash script
extractor.sh that is located in the extracted root folder:
Extracted records or gzipped archives are stored in the sub folder