z/OS Connect - How to trace requests with Instana APM (SMF Type 123 version 2)
Created: | Updated:
Tracing z/OS Connect REST API requests creates SMF type 123 records. The record contains monitoring information like the start and end time of each request, information about the requesting application and information about the system-of-record that processed the request.
SMF 123 records have 2 versions, which stipulates the granularity of the information provided.
Version 1: captures basic data for individual requests.
Version 2: captures enhanced data for each API provider request.
Instana APM sets W3C Trace-Context and Instana specific headers (X-INSTANA-T, X-INSTANA-S) in http call headers. These headers can be used to extract traces that come from applications that are instrumented by Instana APM agents.
Pay careful attention to the sequence if there are any previously configured audit Interceptors. For more information about server.xml and zosconnect_auditInterceptor configuration elements, refer to the IBM z/OS Connect product documentation in the IBM Knowledge Center.
Step 1: Back up the server.xml for the z/OS Connect server that you want to integrate with z/IRIS
Step 2: Configure an audit interceptor
Step 2a: Open the server.xml for editing
The Instana agent sets 4 headers which can be extracted from SMF records. Namely: traceparent and tracestate or X-INSTANA-T and X-INSTANA-S.
If your APM environment is a hybrid mix of Instana and OpenTelemetry agents, follow Step 2b1, otherwise follow Step 2b2.
Step 2b1: Configure Audit Interceptors for requests with tracestate and traceparent in the request headers
W3C headers are recommended for future proofing mainframe observability
<zosconnect_auditInterceptor id="instanaInterceptor" apiProviderSmfVersion="2" apiProviderRequestHeaders="tracestate, traceparent" />
id should be unique in this server.xml file
Step 2b2: Configure Audit Interceptors for requests with X-INSTANA-T and X-INSTANA-S in the request headers
<zosconnect_auditInterceptor id="instanaInterceptor" apiProviderSmfVersion="2" apiProviderRequestHeaders="X-INSTANA-T,X-INSTANA-S" />
Step 3: Enable the Audit Interceptor for this z/OS Connect server
Add the new zosconnect_auditInterceptor id to the interceptorRef attribute of the zosconnect_zosConnectInterceptors or create a new zosconnect_zosConnectInterceptors if one was not previously configured.
<zosconnect_zosConnectInterceptors id="myServerInterceptorList" interceptorRef="instanaInterceptor"/>
Step 4: Refresh the z/OS Connect server
Step 6a: Dynamic refresh
If the updateTrigger="mbean" config element was configured in the server.xml file, a refresh can be dynamically performed to activate the new configuration using the MVS z/OS Connect MODIFY command. If the updateTrigger is not configured, a manual restart will be required (Step 6b)
Example of refreshing the z/OS Connect server configuration
Example of manually stopping and starting the z/OS Connect server
/S BAQSTRT,PARMS='<serverName> [--clean]
Step 5: Verify tracing is activated
Check the logs for confirmation that the audit interceptor configuration has been accepted and activated.
Example z/OS Connect messages.log audit interceptor confirmation
[2/11/21 9:31:29:906 GMT] 000004de om.ibm.zosconnect.audit.smf.internal.AuditSMFInterceptorImpl I BAQR0801I: When active, the audit interceptor zirisMetrics is set to record SMF 123 subtype 1 version 2 records for API provider and version 1 records for API requester. [2/11/21 9:31:30:603 GMT] 000004de om.ibm.zosconnect.audit.smf.internal.AuditSMFInterceptorImpl I BAQR0801I: When active, the audit interceptor openTelemetryInterceptor is set to record SMF 123 subtype 1 version 2 records for API provider and version 1 records for API requester. [2/11/21 9:31:30:668 GMT] 000004de om.ibm.zosconnect.audit.smf.internal.AuditSMFInterceptorImpl I BAQR0803I: Request headers to be recorded by the audit interceptor openTelemetryInterceptor are [tracestate, traceparent]. [2/11/21 9:31:32:883 GMT] 000004de om.ibm.zosconnect.audit.smf.internal.AuditSMFInterceptorImpl I BAQR0801I: When active, the audit interceptor hrAuditInterceptor is set to record SMF 123 subtype 1 version 2 records for API provider and version 1 records for API requester. [2/11/21 9:31:33:584 GMT] 000004de com.ibm.zosconnect.internal.ServiceManagerImpl A BAQR7096I: z/OS Connect EE interceptor list myServerInterceptorList is active at the global level with interceptors: zirisMetrics,openTelemetryInterceptor.