Enable SSL between Kafka and z/IRIS

Created 12 Oct 2022, Updated 17 Oct 2021

Summary

This document will help users who wish to enable an encrypted SSL/TLS connection between Apache Kafka and z/IRIS:

Necessary steps and configuration for the Kafka Cluster.
Necessary steps and configuration for the Kafka Producer (z/IRIS z/OS Client).
Necessary steps and configuration for the Kafka Consumer (z/IRIS IronTap).

Prerequisites

A Linux Distribution
Java 8+ (for keytool)
OpenSSL
A trusted certificate (Certificate Authority - CA) for signing.

Preparations

Create a temporary working directory (e.g. /tmp/ssl/) for the SSL files to be generated
```
mkdir -p /tmp/ssl
```
CODE
All following commands should be executed in that directory
```
cd /tmp/ssl
```
CODE

Kafka Cluster

The following steps should be performed on every involved Kafka Broker of the Kafka Cluster

The following proposed configurations are just illustrative examples. Change the used parameter values according to your needs.

Steps to create and configure the Kafka Broker Keystore and Truststore

Create a Kafka Broker Keystore (e.g. broker1.keystore.jks) with custom alias (e.g. broker1), dname (e.g. CN=broker1) and password (e.g. test1234)
```
keytool -genkey -keystore broker1.keystore.jks -alias broker1 -dname CN=broker1 -keyalg RSA -validity 365 -storepass test1234
```
CODE

Export the unsigned Kafka Broker Certificate (e.g. broker1.unsigned.crt)

keytool -certreq -keystore broker1.keystore.jks -alias broker1 -file broker1.unsigned.crt -storepass test1234

CODE

Get a signed version of the Kafka Broker Certificate (e.g. broker1.crt) by signing it with your trusted Certificate (CA, e.g. ca.crt)

openssl x509 -req -CA ca.crt -CAkey ca.key -in broker1.unsigned.crt -out kafka.broker.crt -days 365 -CAcreateserial -passin pass:test1234

CODE

Import the Certificate of the CA (e.g. ca.crt) with a custom alias (e.g. ca) into the Kafka Broker Keystore (e.g. broker1.keystore.jks)
```
keytool -import -file ca.crt -keystore broker1.keystore.jks -alias ca -storepass test1234 -noprompt
```
CODE
Import the signed Kafka Broker Certificate (e.g. broker1.crt) with the previous used alias (e.g. broker1) into the Kafka Broker Keystore (e.g. broker1.keystore.jks)
```
keytool -import -file broker1.crt -keystore broker1.keystore.jks -alias broker1 -storepass test1234 -noprompt
```
CODE
Import the Certificate of the CA (e.g. ca.crt) with a custom alias (e.g. ca) into the Kafka Broker Truststore (e.g. broker1.truststore.jks), so the brokers can trust it when connecting using SSL.
```
keytool -import -file ca.crt -keystore broker1.truststore.jks -alias ca -storepass test1234 -noprompt
```
CODE

The following commands can be used to double-check the contents of a store

Check the contents of a Kafka Broker Keystore (e.g. broker1.keystore.jks). For every Kafka Broker, there should be two entries.
```
keytool -list -v -keystore broker1.keystore.jks -storepass test1234
```
CODE
- One for the Certificate of the CA
- One for the Kafka Broker itself
Check the contents of a Kafka Broker Truststore. For every Kafka Broker, there should be one entry.
```
keytool -list -v -keystore broker1.truststore.jks -storepass test1234
```
CODE
- One for the Kafka Broker itself

Configure Kafka Broker properties to enable SSL authentication

Modify the following parameters in all Kafka Broker properties files which are used to start up the Kafka Brokers of the Kafka Cluster

listener.security.protocol.map=INTERNAL:SSL,CONTROLLER:SSL,SSL:SSL
listeners=INTERNAL://:9092,CONTROLLER://:9093,SSL://:9094
advertised.listeners=INTERNAL://:9092,SSL://<brokerX-ip>:9094
inter.broker.listener.name=INTERNAL
controller.listener.names=CONTROLLER
process.roles=broker,controller
node.id=<brokerX-id>
controller.quorum.voters=1@<broker1-ip>:9093,<brokerX-id>@<brokerX-ip>:9093
log.dirs=/tmp/kRaft/
security.protocol=SSL
ssl.client.auth=required
ssl.key.password=test1234
ssl.keystore.location=/tmp/ssl/<brokerX>.keystore.jks
ssl.keystore.password=test1234
ssl.truststore.location=/tmp/ssl/<brokerX>.truststore.jks
ssl.truststore.password=test1234
ssl.protocol=TLSv1.2
ssl.endpoint.identification.algorithm=

CODE

Configure the Kafka Confluent Image to enable SSL authentication

Amend or add the following parameters for all involved Kafka Brokers of the Cluster.

The mentioned *.creds files only contain the respective password (in our case test1234) and are located in the directory /tmp/sslwhich is mounted into the container

ports:
 - 9094:9094
volumes:
 - /tmp/ssl:/etc/kafka/secrets
 - /tmp/kafka/use_KRaft.sh:/tmp/kafka/use_KRaft.sh
 environment:
  KAFKA_BROKER_ID: <brokerX-id>
  KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:SSL,CONTROLLER:SSL,SSL:SSL
  KAFKA_ADVERTISED_LISTENERS: INTERNAL://:9092,SSL://<brokerX-ip>:9094
  KAFKA_LISTENERS: INTERNAL://:9092,CONTROLLER://:9093,SSL://:9094
  KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
  KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
  KAFKA_PROCESS_ROLES: broker,controller
  KAFKA_NODE_ID: <brokerX-id>
  KAFKA_CONTROLLER_QUORUM_VOTERS: 1@<broker1-ip>:9093,<brokerX-id>@<brokerX-ip>:9093
  KAFKA_LOG_DIRS: /tmp/kRaft/
  KAFKA_SECURITY_PROTOCOL: SSL
  KAFKA_SSL_CLIENT_AUTH: required
  KAFKA_SSL_KEY_CREDENTIALS: <brokerX>.key.creds
  KAFKA_SSL_KEYSTORE_FILENAME: <brokerX>.keystore.jks
  KAFKA_SSL_KEYSTORE_CREDENTIALS: <brokerX>.keystore.creds
  KAFKA_SSL_TRUSTSTORE_FILENAME: <brokerX>.truststore.jks
  KAFKA_SSL_TRUSTSTORE_CREDENTIALS: <brokerX>.truststore.creds
  KAFKA_SSL_PROTOCOL: TLSv1.2
  KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM:
command: [bash, -c, "/tmp/kafka/use_KRaft.sh && /etc/confluent/docker/run"]

CODE

The command parameter is needed to configure internal scripts before running Kafka in KRaft mode. For that purpose the following script use_KRaft.sh needs to be mounted into the container (at location /tmp/kafka/use_KRaft.sh in this example).

Kafka Producer (z/IRIS z/OS Client)

The following proposed configurations are just illustrative examples. Change the used parameter values according to your needs.

Steps to create and configure the Kafka Producer Keystore and Truststore

Create a z/IRIS z/OS Client Keystore (e.g. ziris.keystore.jks) with custom alias (e.g. ziris), dname (e.g. CN=ziris) and password (e.g. test1234)
```
keytool -genkey -keystore ziris.keystore.jks -alias ziris -dname CN=ziris -keyalg RSA -validity 365 -storepass test1234
```
CODE

Export the unsigned z/IRIS z/OS Client Certificate (e.g. ziris.unsigned.crt)

keytool -certreq -keystore ziris.keystore.jks -alias ziris -file ziris.unsigned.crt -storepass test1234

CODE

Get a signed version of the z/IRIS z/OS Client Certificate (e.g. ziris.crt) by signing it with your trusted Certificate (CA, e.g. ca.crt)
```
openssl x509 -req -CA ca.crt -CAkey ca.key -in ziris.unsigned.crt -out ziris.crt -days 365 -CAcreateserial -passin pass:test1234
```
CODE
Import the Certificate of the CA (e.g. ca.crt) with a custom alias (e.g. ca) into the z/IRIS z/OS Client Keystore (e.g. ziris.keystore.jks)
```
keytool -import -file ca.crt -keystore ziris.keystore.jks -alias ca -storepass test1234 -noprompt
```
CODE
Import the signed z/IRIS z/OS Client Certificate (e.g. ziris.crt) with the previous used alias (e.g. ziris) into the z/IRIS z/OS Client Keystore (e.g. ziris.keystore.jks)
```
keytool -import -file ziris.crt -keystore ziris.keystore.jks -alias ziris -storepass test1234 -noprompt
```
CODE
Import the Certificate of the CA (e.g. ca.crt) with a custom alias (e.g. ca) into the z/IRIS z/OS Client Truststore (e.g. ziris.truststore.jks), so the z/IRIS z/OS Clients can trust it when connecting using SSL.
```
keytool -import -file ca.crt -keystore ziris.truststore.jks -alias ca -storepass test1234 -noprompt
```
CODE

The following commands can be used to double-check the contents of a store

Check the contents of a z/IRIS z/OS Client Keystore (e.g. ziris.keystore.jks). There should be two entries.
```
keytool -list -v -keystore ziris.keystore.jks -storepass test1234
```
CODE
- One for the Certificate of the CA
- One for the z/IRIS z/OS Client itself
Check the contents of a z/IRIS z/OS Client Truststore (e.g. ziris.truststore.jks). There should be one entry.
```
keytool -list -v -keystore ziris.truststore.jks -storepass test1234
```
CODE
- One for the z/IRIS z/OS Client itself

Adjust the z/IRIS z/OS Client configuration to enable SSL authentication

See Configure z/IRIS Clients | Customizing Kafka producer properties to customize the producer properties for the z/IRIS z/OS Client with the following parameters to enable SSL.

security.protocol='SSL'
ssl.protocol='TLSv1.2'
ssl.key.password='test1234'
keystore.location='/tmp/ssl/irontap.keystore.jks'
keystore.password='test1234'
truststore.location='/tmp/ssl/irontap.truststore.jks'
truststore.password='test1234'

CODE

Kafka Consumer (z/IRIS IronTap)

The following proposed configurations are just illustrative examples. Change the used parameter values according to your needs.

Steps to create and configure the Kafka Consumer Keystore and Truststore

Create a z/IRIS Irontap Keystore (e.g. irontap.keystore.jks) with custom alias (e.g. irontap), dname (e.g. CN=irontap) and password (e.g. test1234)
```
keytool -genkey -keystore irontap.keystore.jks -alias irontap -dname CN=irontap -keyalg RSA -validity 365 -storepass test1234
```
CODE

Export the unsigned z/IRIS Irontap Certificate (e.g. irontap.unsigned.crt)

keytool -certreq -keystore irontap.keystore.jks -alias irontap -file irontap.unsigned.crt -storepass test1234

CODE

Get a signed version of the z/IRIS Irontap Certificate (e.g. irontap.crt) by signing it with your trusted Certificate (CA, e.g. ca.crt)
```
openssl x509 -req -CA ca.crt -CAkey ca.key -in irontap.unsigned.crt -out irontap.crt -days 365 -CAcreateserial -passin pass:test1234
```
CODE
Import the Certificate of the CA (e.g. ca.crt) with a custom alias (e.g. ca) into the z/IRIS Irontap Keystore (e.g. ziris.keystore.jks)
```
keytool -import -file ca.crt -keystore irontap.keystore.jks -alias ca -storepass test1234 -noprompt
```
CODE
Import the signed z/IRIS Irontap Certificate (e.g. irontap.crt) with the previous used alias (e.g. irontap) into the z/IRIS Irontap Keystore (e.g. irontap.keystore.jks)
```
keytool -import -file irontap.crt -keystore irontap.keystore.jks -alias irontap -storepass test1234 -noprompt
```
CODE
Import the Certificate of the CA (e.g. ca.crt) with a custom alias (e.g. ca) into the z/IRIS Irontap Truststore (e.g. irontap.truststore.jks), so the z/IRIS Irontap instances can trust it when connecting using SSL.
```
keytool -import -file ca.crt -keystore irontap.truststore.jks -alias ca -storepass test1234 -noprompt
```
CODE

The following commands can be used to double-check the contents of a store

Check the contents of a z/IRIS Irontap Keystore (e.g. irontap.keystore.jks). There should be two entries.
```
keytool -list -v -keystore irontap.keystore.jks -storepass test1234
```
CODE
- One for the Certificate of the CA
- One for the z/IRIS Irontap itself
Check the contents of a z/IRIS Irontap Truststore (e.g. irontap.truststore.jks). There should be one entry.
```
keytool -list -v -keystore irontap.truststore.jks -storepass test1234
```
CODE
- One for the z/IRIS Irontap itself

Extend the IronTap server configuration to enable SSL authentication

Add the following parameters to the IronTap server configuration file of every IronTap server instance to enable SSL.

irontap.sources.kafka.consumer.security.protocol='SSL'
irontap.sources.kafka.consumer.ssl.protocol='TLSv1.2'
irontap.sources.kafka.consumer.ssl.key.password='test1234'
irontap.sources.kafka.consumer.ssl.keystore.location='/tmp/ssl/irontap.keystore.jks'
irontap.sources.kafka.consumer.ssl.keystore.password='test1234'
irontap.sources.kafka.consumer.ssl.truststore.location='/tmp/ssl/irontap.truststore.jks'
irontap.sources.kafka.consumer.ssl.truststore.password='test1234'

CODE

Extend the IronTap container configuration to enable SSL authentication

Amend the following parameters to the docker command of every IronTap server instance to enable SSL.

docker run \
  -- mount type=bind,source=/tmp/ssl,target=/tmp/ssl,readonly \
  mainstorconcept.jfrog.io/ziris-docker-release/irontap:latest-kafka-otel \
  -Dirontap.sources.kafka.consumer.security.protocol='SSL' \
  -Dirontap.sources.kafka.consumer.ssl.protocol='TLSv1.2' \
  -Dirontap.sources.kafka.consumer.ssl.key.password='test1234' \  
  -Dirontap.sources.kafka.consumer.ssl.keystore.location='/tmp/ssl/irontap.keystore.jks' \
  -Dirontap.sources.kafka.consumer.ssl.keystore.password='test1234' \
  -Dirontap.sources.kafka.consumer.ssl.truststore.location='/tmp/ssl/irontap.truststore.jks' \
  -Dirontap.sources.kafka.consumer.ssl.truststore.password='test1234'

CODE