Troubleshooting ziris-instrumentation-extension

Created 19 Jul 2021, Updated 30 Aug 2021

Refer to the following steps to resolve tracing issues related to the OpenTelemetry Instrumentation Extension feature with OpenTelemetry.

Step 1: Check Components

Step 1.a: OpenTelemetry components

Follow the Troubleshooting OpenTelemetry integration article to verify that the relevant OpenTelemetry components are available and implemented correctly.

Step 1.b: ziris-instrumentation-extension

Refer to the ziris-instrumentation-extension versioning to verify that the OpenTelemetry Agent is compatible with the ziris-instrumentation-extension module implemented.

Step 2: Activation

Refer to Activate ziris-instrumentation-extension to verify that the z/IRIS Instrumentation Extension module has been activated correctly.

Step 3: Configuration

Refer to Configure ziris-instrumentation-extensionto verify that the z/IRIS Instrumentation Extension has been configured correctly.

If -Dmainstorconcept.ziris.jdbc.field has been configured, it must be set to one of the provided values. Incorrect values will result in the following warning message will be logged and the z/IRIS instrumentation will be disabled.

[otel.javaagent 2021-07-14 20:10:38:457 +0200] [main] WARN com.mainstorconcept.ziris.instrumentation.jdbc.helper.ZirisConfig - JDBC field 'ApplicationHostName' is not supported, z/IRIS instrumentation will be disabled.

CODE

Step 4: Logging

Verify that the instrumentation works correctly using logging:

Verify that

Assuming that your command line looks similar to this:

java -javaagent:path/to/opentelemetry-javaagent-all.jar \
     -Dotel.javaagent.extensions=path/to/ziris-instrumentation-extension.jar \
     -jar path/to/jar/application/that/should/be/instrumented.jar

CODE

Adjust it to turn on the agent’s internal debugging:

java -javaagent:path/to/opentelemetry-javaagent-all.jar \
     -Dotel.javaagent.debug=true \
     -Dotel.javaagent.extensions=path/to/ziris-instrumentation-extension.jar 2>&1 | grep ziris-jdbc

CODE

The following output verifies that the agent actually loads the z/IRIS OpenTelemetry Java Instrumentation Extension:

[otel.javaagent 2021-07-08 21:03:04:959 +0200] [main] DEBUG io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationLoader - Loading instrumentation ziris-jdbc [class com.mainstorconcept.ziris.instrumentation.ZirisJdbcInstrumentationModule]

CODE

(Optional) Prepare OpenTelemetry (Contrib) Collector

An OpenTelemetry (Contrib) Collector can be configured and used as an easy option to enhance debug and troubleshooting capabilities. Any other option which provides an OTLP endpoint and has such capabilities can be used as well (like Jaeger).

Add a logging exporter to the OpenTelemetry (Contrib) Collector configuration file to enable debug logging. Example:

Remember to set the logging exporter in the service section.

exporters:
  logging:
    loglevel: debug

service:
  pipelines:
    traces:
      exporters: [logging]

CODE

(Re-)start the OpenTelemetry (Contrib) Collector to activate debug logging. Example:

systemctl restart otel-collector            # OpenTelemetry Collector 
systemctl restart otel-contrib-collector    # OpenTelemetry Contrib Collector

BASH

Confirm that the OpenTelemetry (Contrib) Collector is running. Example:

systemctl is-active otel-contrib-collector  # OpenTelemetry Collector 
systemctl is-active otel-contrib-collector  # OpenTelemetry Contrib Collector

TEXT

Expected result is that OpenTelemetry (Contrib) Collector is running. Example output:

active

CODE

Now the OpenTelemetry (Contrib) Collector logs can be reviewed. Example:

journalctl --unit=otel-collector            # OpenTelemetry Collector 
journalctl --unit=otel-contrib-collector    # OpenTelemetry Contrib Collector
less /var/log/syslog | grep otel            # Log location for Debian-based systems (like Ubuntu)
less /var/log/messages | grep otel          # Log location for Red Hat-based systems (like CentOS)

CODE

Prepare z/IRIS Irontap logging

Open a command prompt, switch to the root folder of z/IRIS IronTap and set the logging level of its elements to trace in the provided log4j2.xml file. Example:

cd path/to/irontap
sed -i s/'name="com.mainstorconcept" level="trace"'/'name="com.mainstorconcept" level="info"'/ log4j2.xml

CODE

Confirm application instrumentation extension

This example uses a configured OpenTelemetry (Contrib) Collector for debug and troubleshooting capabilities. Any other option which provides an OTLP endpoint and has such capabilities can be used as well (like Jaeger).

Open a command prompt, switch to the root folder of z/IRIS IronTap and start z/IRIS IronTap. Example:

cd path/to/irontap
./run.sh

CODE

Assuming that your command line looks similar to this for starting the OpenTelemetry Agent in conjunction with the z/IRIS OpenTelemetry Java Instrumentation Extension in a new command prompt:

java -javaagent:path/to/opentelemetry-javaagent-all.jar \
     -Dotel.javaagent.extensions=path/to/ziris-instrumentation-extension.jar \
     -jar path/to/jar/application/that/should/be/instrumented.jar

CODE

Change it so that the previously configured uses OpenTelemetry (Contrib) Collector, or any other option with debug capabilities, as OTLP endpoint:

java -javaagent:path/to/opentelemetry-javaagent-all.jar \
     -Dotel.javaagent.extensions=path/to/ziris-instrumentation-extension.jar \
     -Dotel.exporter.otlp.endpoint=http://127.0.0.1:4317 \
     -jar path/to/jar/application/that/should/be/instrumented.jar

CODE

Verify that the used OTLP endpoint receives traces. In case of the OpenTelemetry (Contrib) Collector, review its logs in a new command prompt:

journalctl --unit=otel-collector | grep -A 2 "Trace ID"                # OpenTelemetry Collector
journalctl --unit=otel-contrib-collector | grep -A 2 "Trace ID"        # OpenTelemetry Contrib Collector
less /var/log/syslog | grep otel | grep -A 2 "Trace ID"                # Log location for Debian-based systems (like Ubuntu)
less /var/log/messages | grep otel | grep -A 2 "Trace ID"              # Log location for Red Hat-based systems (like CentOS)

CODE

Various traces and spans should be visible in the logs. Example output:

Jul 08 21:24:36 repo01.mainstorconcept.com otelcontribcol[712333]:     Trace ID       : bf3e9d28f38060cae2d7fdd8898d7782
Jul 08 21:24:36 repo01.mainstorconcept.com otelcontribcol[712333]:     Parent ID      :
Jul 08 21:24:36 repo01.mainstorconcept.com otelcontribcol[712333]:     ID             : ab31ddd95a5804e9
--
Jul 08 21:24:36 repo01.mainstorconcept.com otelcontribcol[712333]:     Trace ID       : 392cc07218b824cc5120253a5606cb9c
Jul 08 21:24:36 repo01.mainstorconcept.com otelcontribcol[712333]:     Parent ID      :
Jul 08 21:24:36 repo01.mainstorconcept.com otelcontribcol[712333]:     ID             : 0bd5431880b12adc

CODE

Open a command prompt, switch to the root folder of z/IRIS IronTap and review the z/IRIS IronTap logs and confirm the traces. Example:

cd path/to/irontap
less logs/irontapInfo.log | grep -Po 'trace=.*,span=.*'

CODE

Corresponding trace and span ids will be logged by IronTap. Example output:

trace=bf3e9d28f38060cae2d7fdd8898d7782,span=ab31ddd95a5804e9
trace=392cc07218b824cc5120253a5606cb9c,span=0bd5431880b12adc

CODE