Troubleshooting ziris-instrumentation-extension
Created , Updated
Refer to the following steps to resolve tracing issues related to the OpenTelemetry Instrumentation Extension feature with OpenTelemetry.
Step 1: Check Components
Step 1.a: OpenTelemetry components
Follow the Troubleshooting OpenTelemetry integration article to verify that the relevant OpenTelemetry components are available and implemented correctly.
Step 1.b: ziris-instrumentation-extension
Refer to the ziris-instrumentation-extension versioning to verify that the OpenTelemetry Agent is compatible with the ziris-instrumentation-extension module implemented.
Step 2: Activation
Refer to Activate ziris-instrumentation-extension to verify that the z/IRIS Instrumentation Extension module has been activated correctly.
Step 3: Configuration
Refer to Configure ziris-instrumentation-extensionto verify that the z/IRIS Instrumentation Extension has been configured correctly.
If -Dmainstorconcept.ziris.jdbc.field
has been configured, it must be set to one of the provided values. Incorrect values will result in the following warning message will be logged and the z/IRIS instrumentation will be disabled.
[otel.javaagent 2021-07-14 20:10:38:457 +0200] [main] WARN com.mainstorconcept.ziris.instrumentation.jdbc.helper.ZirisConfig - JDBC field 'ApplicationHostName' is not supported, z/IRIS instrumentation will be disabled.
Step 4: Logging
Verify that the instrumentation works correctly using logging:
Assuming that your command line looks similar to this:
java -javaagent:path/to/opentelemetry-javaagent-all.jar \
-Dotel.javaagent.extensions=path/to/ziris-instrumentation-extension.jar \
-jar path/to/jar/application/that/should/be/instrumented.jar
Adjust it to turn on the agent’s internal debugging:
java -javaagent:path/to/opentelemetry-javaagent-all.jar \
-Dotel.javaagent.debug=true \
-Dotel.javaagent.extensions=path/to/ziris-instrumentation-extension.jar 2>&1 | grep ziris-jdbc
The following output verifies that the agent actually loads the z/IRIS OpenTelemetry Java Instrumentation Extension:
[otel.javaagent 2021-07-08 21:03:04:959 +0200] [main] DEBUG io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationLoader - Loading instrumentation ziris-jdbc [class com.mainstorconcept.ziris.instrumentation.ZirisJdbcInstrumentationModule]
An OpenTelemetry (Contrib) Collector can be configured and used as an easy option to enhance debug and troubleshooting capabilities. Any other option which provides an OTLP endpoint and has such capabilities can be used as well (like Jaeger).
Add a logging exporter to the OpenTelemetry (Contrib) Collector configuration file to enable debug logging. Example:
Remember to set the logging exporter in the service section.
exporters:
logging:
loglevel: debug
service:
pipelines:
traces:
exporters: [logging]
(Re-)start the OpenTelemetry (Contrib) Collector to activate debug logging. Example:
systemctl restart otel-collector # OpenTelemetry Collector
systemctl restart otel-contrib-collector # OpenTelemetry Contrib Collector
Confirm that the OpenTelemetry (Contrib) Collector is running. Example:
systemctl is-active otel-contrib-collector # OpenTelemetry Collector
systemctl is-active otel-contrib-collector # OpenTelemetry Contrib Collector
Expected result is that OpenTelemetry (Contrib) Collector is running. Example output:
active
Now the OpenTelemetry (Contrib) Collector logs can be reviewed. Example:
journalctl --unit=otel-collector # OpenTelemetry Collector
journalctl --unit=otel-contrib-collector # OpenTelemetry Contrib Collector
less /var/log/syslog | grep otel # Log location for Debian-based systems (like Ubuntu)
less /var/log/messages | grep otel # Log location for Red Hat-based systems (like CentOS)
Open a command prompt, switch to the root folder of z/IRIS IronTap and set the logging level of its elements to trace in the provided log4j2.xml file. Example:
cd path/to/irontap
sed -i s/'name="com.mainstorconcept" level="trace"'/'name="com.mainstorconcept" level="info"'/ log4j2.xml
This example uses a configured OpenTelemetry (Contrib) Collector for debug and troubleshooting capabilities. Any other option which provides an OTLP endpoint and has such capabilities can be used as well (like Jaeger).
Open a command prompt, switch to the root folder of z/IRIS IronTap and start z/IRIS IronTap. Example:
cd path/to/irontap
./run.sh
Assuming that your command line looks similar to this for starting the OpenTelemetry Agent in conjunction with the z/IRIS OpenTelemetry Java Instrumentation Extension in a new command prompt:
java -javaagent:path/to/opentelemetry-javaagent-all.jar \
-Dotel.javaagent.extensions=path/to/ziris-instrumentation-extension.jar \
-jar path/to/jar/application/that/should/be/instrumented.jar
Change it so that the previously configured uses OpenTelemetry (Contrib) Collector, or any other option with debug capabilities, as OTLP endpoint:
java -javaagent:path/to/opentelemetry-javaagent-all.jar \
-Dotel.javaagent.extensions=path/to/ziris-instrumentation-extension.jar \
-Dotel.exporter.otlp.endpoint=http://127.0.0.1:4317 \
-jar path/to/jar/application/that/should/be/instrumented.jar
Verify that the used OTLP endpoint receives traces. In case of the OpenTelemetry (Contrib) Collector, review its logs in a new command prompt:
journalctl --unit=otel-collector | grep -A 2 "Trace ID" # OpenTelemetry Collector
journalctl --unit=otel-contrib-collector | grep -A 2 "Trace ID" # OpenTelemetry Contrib Collector
less /var/log/syslog | grep otel | grep -A 2 "Trace ID" # Log location for Debian-based systems (like Ubuntu)
less /var/log/messages | grep otel | grep -A 2 "Trace ID" # Log location for Red Hat-based systems (like CentOS)
Various traces and spans should be visible in the logs. Example output:
Jul 08 21:24:36 repo01.mainstorconcept.com otelcontribcol[712333]: Trace ID : bf3e9d28f38060cae2d7fdd8898d7782
Jul 08 21:24:36 repo01.mainstorconcept.com otelcontribcol[712333]: Parent ID :
Jul 08 21:24:36 repo01.mainstorconcept.com otelcontribcol[712333]: ID : ab31ddd95a5804e9
--
Jul 08 21:24:36 repo01.mainstorconcept.com otelcontribcol[712333]: Trace ID : 392cc07218b824cc5120253a5606cb9c
Jul 08 21:24:36 repo01.mainstorconcept.com otelcontribcol[712333]: Parent ID :
Jul 08 21:24:36 repo01.mainstorconcept.com otelcontribcol[712333]: ID : 0bd5431880b12adc
Open a command prompt, switch to the root folder of z/IRIS IronTap and review the z/IRIS IronTap logs and confirm the traces. Example:
cd path/to/irontap
less logs/irontapInfo.log | grep -Po 'trace=.*,span=.*'
Corresponding trace and span ids will be logged by IronTap. Example output:
trace=bf3e9d28f38060cae2d7fdd8898d7782,span=ab31ddd95a5804e9
trace=392cc07218b824cc5120253a5606cb9c,span=0bd5431880b12adc