We suggest using:
- an odd number of Zookeeper instances with a minimum of 3 instances.
- at least 3 Kafka Brokers with a replication factor of 3 deployed on (physically) separate machines. This will exploit the replication factor effectively and ensure high availability and resilience.
For the purposes of z/IRIS, a Kafka topic is needed to enable producing and consuming of SMF data. To determine a suitable number of partitions for the topic the following should be considered:
- The number of partitions should be a multiple of the number of Kafka Brokers to achieve proper Load Balancing between them.
- Because a partition can only be assigned to one consumer in a consumer group, more partitions increases:
- the possibility to leverage more Kafka brokers in a cluster,
- and scale capabilities.
- Fewer partitions means that fewer files are opened on the Kafka cluster for Zookeeper to manage.
Taking this into consideration, we recommend using around 15 Partitions for the topic. Select a suitable topic name, e.g. smf, and create the topic with the help of the Kafka CLI:
kafka-topics.sh --bootstrap-server <kafka-ip>:<kafka-port> --topic smf --partitions 15 --replication-factor 3 --create
To further reduce the risk of data loss, we suggest 2 In-Sync Replicas, which can be configured after creating the topic smf by:
kafka-configs.sh --zookeeper <zookeeper-ip>:<zookeeper-port> --entity-type topics --entity-name smf --add-config min.insync.replicas=2 --alter
Once z/IRIS IronTap is set up to use the Kafka Cluster, we highly recommend using Apache Kafka's feature named Quotas, which protects against the possibility of monopolizing resources and causing network saturation. To achieve this, Kafka needs to identify IronTap (i.e. its Kafka consumers) in the Kafka cluster. This can be done by specifying a unique, logical application name (e.g. smf-consumer) for the client.id property in the IronTap application.conf file. Afterwards, Apache Kafka's Quotas can be set dynamically via the Kafka CLI. We recommend limiting the network bandwidth to around ~50 MB/s for IronTap:
kafka-configs.sh --zookeeper <zookeeper-ip>:<zookeeper-port> --entity-type clients --entity-name <client-id> --add-config 'consumer_byte_rate=52428800' --alter