Troubleshooting ziris-instrumentation-extension

Created 19 Jul 2021, Updated 26 Jun 2023

Refer to the following steps to resolve tracing issues related to the OpenTelemetry Instrumentation Extension feature with OpenTelemetry.

Step 1: Check Components

Step 1.a: OpenTelemetry components

Follow the Troubleshooting OpenTelemetry integration article to verify that the relevant OpenTelemetry components are available and implemented correctly.

Step 1.b: ziris-instrumentation-extension

Refer to the ziris-instrumentation-extension versioning to verify that the OpenTelemetry Agent is compatible with the ziris-instrumentation-extension module implemented.

Step 2: Activation

Refer to Activate ziris-instrumentation-extension to verify that the z/IRIS Instrumentation Extension module has been activated correctly.

Step 3: Configuration

Refer to Configure ziris-instrumentation-extensionto verify that the z/IRIS Instrumentation Extension has been configured correctly.

If -Dmainstorconcept.ziris.jdbc.field has been configured, it must be set to one of the provided values. Incorrect values will result in the following warning message will be logged and the z/IRIS instrumentation will be disabled.

CODE

[otel.javaagent 2021-07-14 20:10:38:457 +0200] [main] WARN com.mainstorconcept.ziris.instrumentation.jdbc.helper.ZirisConfig - JDBC field 'ApplicationHostName' is not supported, z/IRIS instrumentation will be disabled.

Step 4: Logging

Verify that the instrumentation works correctly using logging:

Verify that

Assuming that your command line looks similar to this:

CODE

java -javaagent:path/to/opentelemetry-javaagent-all.jar \
     -Dotel.javaagent.extensions=path/to/ziris-instrumentation-extension.jar \
     -jar path/to/jar/application/that/should/be/instrumented.jar

Adjust it to turn on the agent’s internal debugging:

CODE

java -javaagent:path/to/opentelemetry-javaagent-all.jar \
     -Dotel.javaagent.debug=true \
     -Dotel.javaagent.extensions=path/to/ziris-instrumentation-extension.jar 2>&1 | grep ziris-jdbc

The following output verifies that the agent actually loads the z/IRIS OpenTelemetry Java Instrumentation Extension:

CODE

[otel.javaagent 2021-07-08 21:03:04:959 +0200] [main] DEBUG io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationLoader - Loading instrumentation ziris-jdbc [class com.mainstorconcept.ziris.instrumentation.ZirisJdbcInstrumentationModule]

(Optional) Prepare OpenTelemetry (Contrib) Collector

An OpenTelemetry (Contrib) Collector can be configured and used as an easy option to enhance debug and troubleshooting capabilities. Any other option which provides an OTLP endpoint and has such capabilities can be used as well (like Jaeger).

Add a logging exporter to the OpenTelemetry (Contrib) Collector configuration file to enable debug logging. Example:

Remember to set the logging exporter in the service section.

CODE

exporters:
  logging:
    loglevel: debug

service:
  pipelines:
    traces:
      exporters: [logging]

(Re-)start the OpenTelemetry (Contrib) Collector to activate debug logging. Example:

BASH

systemctl restart otel-collector            # OpenTelemetry Collector 
systemctl restart otel-contrib-collector    # OpenTelemetry Contrib Collector

Confirm that the OpenTelemetry (Contrib) Collector is running. Example:

TEXT

systemctl is-active otel-contrib-collector  # OpenTelemetry Collector 
systemctl is-active otel-contrib-collector  # OpenTelemetry Contrib Collector

Expected result is that OpenTelemetry (Contrib) Collector is running. Example output:

CODE

active

Now the OpenTelemetry (Contrib) Collector logs can be reviewed. Example:

CODE

journalctl --unit=otel-collector            # OpenTelemetry Collector 
journalctl --unit=otel-contrib-collector    # OpenTelemetry Contrib Collector
less /var/log/syslog | grep otel            # Log location for Debian-based systems (like Ubuntu)
less /var/log/messages | grep otel          # Log location for Red Hat-based systems (like CentOS)

Prepare z/IRIS Irontap logging

Open a command prompt, switch to the root folder of z/IRIS IronTap and set the logging level of its elements to trace in the provided log4j2.xml file. Example:

CODE

cd path/to/irontap
sed -i s/'name="com.mainstorconcept" level="trace"'/'name="com.mainstorconcept" level="info"'/ log4j2.xml

Confirm application instrumentation extension

This example uses a configured OpenTelemetry (Contrib) Collector for debug and troubleshooting capabilities. Any other option which provides an OTLP endpoint and has such capabilities can be used as well (like Jaeger).

Open a command prompt, switch to the root folder of z/IRIS IronTap and start z/IRIS IronTap. Example:

CODE

cd path/to/irontap
./run.sh

Assuming that your command line looks similar to this for starting the OpenTelemetry Agent in conjunction with the z/IRIS OpenTelemetry Java Instrumentation Extension in a new command prompt:

CODE

java -javaagent:path/to/opentelemetry-javaagent-all.jar \
     -Dotel.javaagent.extensions=path/to/ziris-instrumentation-extension.jar \
     -jar path/to/jar/application/that/should/be/instrumented.jar

Change it so that the previously configured uses OpenTelemetry (Contrib) Collector, or any other option with debug capabilities, as OTLP endpoint:

CODE

java -javaagent:path/to/opentelemetry-javaagent-all.jar \
     -Dotel.javaagent.extensions=path/to/ziris-instrumentation-extension.jar \
     -Dotel.exporter.otlp.endpoint=http://127.0.0.1:4317 \
     -jar path/to/jar/application/that/should/be/instrumented.jar

Verify that the used OTLP endpoint receives traces. In case of the OpenTelemetry (Contrib) Collector, review its logs in a new command prompt:

CODE

journalctl --unit=otel-collector | grep -A 2 "Trace ID"                # OpenTelemetry Collector
journalctl --unit=otel-contrib-collector | grep -A 2 "Trace ID"        # OpenTelemetry Contrib Collector
less /var/log/syslog | grep otel | grep -A 2 "Trace ID"                # Log location for Debian-based systems (like Ubuntu)
less /var/log/messages | grep otel | grep -A 2 "Trace ID"              # Log location for Red Hat-based systems (like CentOS)

Various traces and spans should be visible in the logs. Example output:

CODE

Jul 08 21:24:36 otelcontribcol[712333]:     Trace ID       : bf3e9d28f38060cae2d7fdd8898d7782
Jul 08 21:24:36 otelcontribcol[712333]:     Parent ID      :
Jul 08 21:24:36 otelcontribcol[712333]:     ID             : ab31ddd95a5804e9
--
Jul 08 21:24:36 otelcontribcol[712333]:     Trace ID       : 392cc07218b824cc5120253a5606cb9c
Jul 08 21:24:36 otelcontribcol[712333]:     Parent ID      :
Jul 08 21:24:36 otelcontribcol[712333]:     ID             : 0bd5431880b12adc

Open a command prompt, switch to the root folder of z/IRIS IronTap and review the z/IRIS IronTap logs and confirm the traces. Example:

CODE

cd path/to/irontap
less logs/irontapInfo.log | grep -Po 'trace=.*,span=.*'

Corresponding trace and span ids will be logged by IronTap. Example output:

CODE

trace=bf3e9d28f38060cae2d7fdd8898d7782,span=ab31ddd95a5804e9
trace=392cc07218b824cc5120253a5606cb9c,span=0bd5431880b12adc