z/OS Connect - How to trace requests with OpenTelemetry (SMF Type 123 version 2)

Created: 12 Apr 2021 | Updated: 28 Sep 2023

Tracing z/OS Connect REST API processing creates SMF type 123 records. The records contain monitoring information like the start and end time of each request, information about the requesting application call and information about the system-of-record that processed the request.

SMF 123 records have 2 versions, which stipulates the granularity of the information provided.

Version 1: captures basic data for individual requests.
Version 2: captures enhanced data for each API provider request.

OpenTelemetry uses W3C Trace-Context in the REST API headers call. These trace headers can be used to selectively extract traces that come from applications using the OpenTelemetry specification for tracing observability.

Pay careful attention to the sequence if there are any previously configured auditInterceptors. For more information about server.xml and zosconnect_auditInterceptor configuration elements, refer to the IBM z/OS Connect product documentation in the IBM Knowledge Center.

Step 1: Back up the server.xml for the z/OS Connect server that you want to integrate with z/IRIS

Step 2: Configure an audit interceptor

Click to expand instructions ...

Step 2a: Open the server.xml for editing

Step 2b: Configure auditing for requests with tracestate and traceparent in the request headers

XML

<zosconnect_auditInterceptor id="openTelemetryInterceptor"
     sequence="3" apiProviderSmfVersion="2"
     apiProviderRequestHeaders="tracestate, traceparent" />

id should be unique in this server.xml file

Step 3: Enable the audit interceptor for this z/OS Connect server

Click to expand instructions ...

Add the new zosconnect_auditInterceptor id to the interceptorRef attribute of the zosconnect_zosConnectInterceptors or create a new zosconnect_zosConnectInterceptors if one was not previously configured.

XML

<zosconnect_zosConnectInterceptors id="myServerInterceptorList"
      interceptorRef="zirisMetrics, openTelemetryInterceptor"/>

Step 4: Refresh the z/OS Connect server

Click to expand instructions ...

Step 6a: Dynamic refresh

If the updateTrigger="mbean" config element was configured in the server.xml file, a refresh can be dynamically performed to activate the new configuration using the MVS z/OS Connect MODIFY command. If the updateTrigger is not configured, a manual restart will be required (Step 6b)

Example z/OS Connect refresh command

TEXT

/F JOBNAME,zcon,refresh

Step 6b: Stop and start the z/OS Connect

CODE

/P JOBNAME

CODE

/S BAQSTRT,PARMS='<serverName> [--clean]

Step 5: Verify tracing is activated

Check the logs for confirmation that the audit interceptor configuration has been accepted and activated.

Example z/OS Connect messages.log audit interceptor confirmation

CODE

[2/11/21 9:31:29:906 GMT] 000004de om.ibm.zosconnect.audit.smf.internal.AuditSMFInterceptorImpl I BAQR0801I: When active, the audit interceptor zirisMetrics is set to record SMF 123 subtype 1 version 2 records for API provider and version 1 records for API requester.
[2/11/21 9:31:30:603 GMT] 000004de om.ibm.zosconnect.audit.smf.internal.AuditSMFInterceptorImpl I BAQR0801I: When active, the audit interceptor openTelemetryInterceptor is set to record SMF 123 subtype 1 version 2 records for API provider and version 1 records for API requester.
[2/11/21 9:31:30:668 GMT] 000004de om.ibm.zosconnect.audit.smf.internal.AuditSMFInterceptorImpl I BAQR0803I: Request headers to be recorded by the audit interceptor openTelemetryInterceptor are [tracestate, traceparent].
[2/11/21 9:31:32:883 GMT] 000004de om.ibm.zosconnect.audit.smf.internal.AuditSMFInterceptorImpl I BAQR0801I: When active, the audit interceptor hrAuditInterceptor is set to record SMF 123 subtype 1 version 2 records for API provider and version 1 records for API requester.
[2/11/21 9:31:33:584 GMT] 000004de com.ibm.zosconnect.internal.ServiceManagerImpl               A BAQR7096I: z/OS Connect EE interceptor list myServerInterceptorList is active at the global level with interceptors: zirisMetrics,openTelemetryInterceptor.