Created: | Updated:

Tracing z/OS Connect REST API requests creates SMF type 123 records. The record contains monitoring information like the start and end time of each request, information about the requesting application and information about the system-of-record that processed the request.

SMF 123 records have 2 versions, which stipulates the granularity of the information provided.

  • Version 1: captures basic data for individual requests.

  • Version 2: captures enhanced data for each API provider request.

Instana APM sets W3C Trace-Context and Instana specific headers (X-INSTANA-T, X-INSTANA-S) in http call headers. These headers can be used to extract traces that come from applications that are instrumented by Instana APM agents.

Pay careful attention to the sequence if there are any previously configured audit Interceptors. For more information about server.xml and zosconnect_auditInterceptor configuration elements, refer to the IBM z/OS Connect product documentation in the IBM Knowledge Center.

Step 1: Back up the server.xml for the z/OS Connect server that you want to integrate with z/IRIS

Step 2: Configure an audit interceptor

Click to expand instructions ...

Step 2a: Open the server.xml for editing

The Instana agent sets 4 headers which can be extracted from SMF records. Namely: traceparent and tracestate or X-INSTANA-T and X-INSTANA-S.

If your APM environment is a hybrid mix of Instana and OpenTelemetry agents, follow Step 2b1, otherwise follow Step 2b2.

Step 2b1: Configure Audit Interceptors for requests with tracestate and traceparent in the request headers

W3C headers are recommended for future proofing mainframe observability

<zosconnect_auditInterceptor id="instanaInterceptor"
     apiProviderSmfVersion="2"
     apiProviderRequestHeaders="tracestate, traceparent" />
XML

id should be unique in this server.xml file

OR

Step 2b2: Configure Audit Interceptors for requests with X-INSTANA-T and X-INSTANA-S in the request headers

<zosconnect_auditInterceptor id="instanaInterceptor"
     apiProviderSmfVersion="2"
     apiProviderRequestHeaders="X-INSTANA-T,X-INSTANA-S" />
XML

Step 3: Enable the Audit Interceptor for this z/OS Connect server

Click to expand instructions ...

Add the new zosconnect_auditInterceptor id to the interceptorRef attribute of the zosconnect_zosConnectInterceptors or create a new zosconnect_zosConnectInterceptors if one was not previously configured.

<zosconnect_zosConnectInterceptors id="myServerInterceptorList"
      interceptorRef="instanaInterceptor"/>
XML

Step 4: Refresh the z/OS Connect server

Click to expand instructions ...

Step 6a: Dynamic refresh

If the updateTrigger="mbean" config element was configured in the server.xml file, a refresh can be dynamically performed to activate the new configuration using the MVS z/OS Connect MODIFY command. If the updateTrigger is not configured, a manual restart will be required (Step 6b)

Example of refreshing the z/OS Connect server configuration
/F JOBNAME,zcon,refresh
TEXT
Example of manually stopping and starting the z/OS Connect server
/P JOBNAME
CODE 
/S BAQSTRT,PARMS='<serverName> [--clean]
CODE

Step 5: Verify tracing is activated

Check the logs for confirmation that the audit interceptor configuration has been accepted and activated.

Example z/OS Connect messages.log audit interceptor confirmation
[2/11/21 9:31:29:906 GMT] 000004de om.ibm.zosconnect.audit.smf.internal.AuditSMFInterceptorImpl I BAQR0801I: When active, the audit interceptor zirisMetrics is set to record SMF 123 subtype 1 version 2 records for API provider and version 1 records for API requester.
[2/11/21 9:31:30:603 GMT] 000004de om.ibm.zosconnect.audit.smf.internal.AuditSMFInterceptorImpl I BAQR0801I: When active, the audit interceptor openTelemetryInterceptor is set to record SMF 123 subtype 1 version 2 records for API provider and version 1 records for API requester.
[2/11/21 9:31:30:668 GMT] 000004de om.ibm.zosconnect.audit.smf.internal.AuditSMFInterceptorImpl I BAQR0803I: Request headers to be recorded by the audit interceptor openTelemetryInterceptor are [tracestate, traceparent].
[2/11/21 9:31:32:883 GMT] 000004de om.ibm.zosconnect.audit.smf.internal.AuditSMFInterceptorImpl I BAQR0801I: When active, the audit interceptor hrAuditInterceptor is set to record SMF 123 subtype 1 version 2 records for API provider and version 1 records for API requester.
[2/11/21 9:31:33:584 GMT] 000004de com.ibm.zosconnect.internal.ServiceManagerImpl               A BAQR7096I: z/OS Connect EE interceptor list myServerInterceptorList is active at the global level with interceptors: zirisMetrics,openTelemetryInterceptor.
CODE