Page tree
Skip to end of metadata
Go to start of metadata


  • Bypass known Instana UI search limitations, specifically the Analyze view, by using the REST API to ensure all spans delivered by z/IRIS are available in the Instana Backend
  • Detailed analysis of the delivered spans
    • Are the span identifiers (traceId, parentId, spanId) available in the back-end
    • If not, which identifiers could not be found.

Background and strategic fit

Some traces are invisible in the Instana UI, although the z/IRIS IronTap component reports a successful upload of all valid Instana  spans.

This script intends to clarify how many mainframe related traces cannot be located and provides information about the missing traces or calls: traceId, parentId or spanId.

This script was written in Python.


  1. A linux system to execute the script

  2. IronTap dump files with the subsequent structure:"YYYY-MM-DDTHH.MM.SS.NNN+HH.MM-HttpResponseCode.json" (e.g.: 2019-04-11T10.40.38.573+02.00-204.json)
    Example of file content:

          "name":"SQL related IFCIDs",
                   "Aggregated smf overview":[  
                         "Total records":1
                         "IFCID 3 - Accounting Data Collection":[  
                               "Instrumentation Data":[  
                                  "Acctg. Start Time:2019-04-11T08:40:44.341407",
                                  "Acctg. End Time:2019-04-11T08:40:44.358031",
                                  "Response Time:16624 micros",
                                  "CPU Start Time:26990 micros",
                                  "CPU End Time:43464 micros",
                                  "Acctg. Reason:DDF Type 2 Becoming Inactive",
                                  "Network Id:",
                                  "2Phase Commit Requests:1",
                                  "Abort Requests:0",
                                  "SU Conversion Factor:12497",
                                  "Workload Manager Service Class Name:DB2SSYST"
                               "DDF Data":[  
                                  "Threads to roll into QLAC Section:1",
                                  "Remote Location Name:::",
                                  "Remote Product ID:JCC04210",
                                  "Msgs Sent to Remote Site:3",
                                  "Msgs Recv. from Remote Site:3",
                                  "SQL Stmts Recv. from Remote Site:3",
                                  "Bytes Sent to Remote Site:578",
                                  "Bytes Recv. from Remote Site:456",
                                  "Rows Sent to Remote Site:1",
                                  "Block Fetch Blocks Transmitted:2",
                                  "Single/Two Phase Commits Recv.:1"
                               "Lock Usage Data":[  
                                  "Resource Limit Id:",
                                  "Lock Requests:3",
                                  "Unlock Requests:1",
                                  "Claim Requests:2"
                               "MVS & DDF  Data":[  
                                  "Product Version:04",
                                  "Product Release:21",
                                  "Product Modification Level:0",
                                  "JCC Client Platform:",
                                  "JCC Client Appl. Name:db2jcc_application",
                                  "JCC Client Auth Id:insusr1",
                                  "JCC Account Suffix:JCC04210192.168.10.53"
                               "SQL Data":[  
                                  "Byte Flag:",
                                  "Cache Copied Prepared Stmt Reqs.:1",
                                  "Rows Fetched:1"
                               "Buffer Manager Data":[  
                                  "Get Page Requests:34",
                                  "Dynamic Prefetch Requested:3"
                   "luName":"D0BF    ",
                   "instance":"DALLASC         ",
                         "applicationName":"db2jcc_application  ",
                         "authId":"insusr1 "

User interaction and design

The script uses the Instana REST API to compare the traces from the file with those displayed in the Instana Analyze View.

API Calls used:

  • Trace Detail: {basic_instana_url}/api/application-monitoring/analyze/traces;id={id} (for On-Prem instance v1.147.536 or earlier) or  {basic_instana_url}/api/application-monitoring/analyze/traces/{id} (for SaaS instance v1.151.124 or earlier)→ gives us a JSON with all information about this trace and its child spans

How does the Script work?

A zip-file called: is provided

Choose a directory as the location to unzip the .zip file content.

Script name:


Before executing the script, please verify:

  1. Is Python installed? → If not, execute: apt-get install python3.6 (or later)
  2. Is pip installed? → If not, execute: apt-get install python3-pip
  3. If both is available, check if the module "requests" is installed (you can check this by going into the python prompt and use the command help("modules") to get a list of all installed modules or help("<module_name>") to search for a specific module) → If not, execute: python3.6 -m pip install requests

Once the above prerequisites have been met, execute the script.

The subsequent syntax ensures a successful execution of the script:

python3.6 "<" -p "<path_to_IronTap_json_folder>" -b "<basic_instana_url>" -t '<API_token>' -i "<instana_instance>"


The single quotation marks are mandatory: '<API_token>' .


python3.6 -p "/opt/json" -b "" -t '12345LKZWOOPLK' -i "on-prem"


  • -p "<path_to_IronTap_json_folder>" → this has to be an absolute path to the "json"-folder where the debug files are present.
  • -b "<basic_instana_url>" → enter the URL to your instana on-prem or SaaS instance
  • -t '<API_token>' → any API token to authorize REST API calls
  • -i "<instana_instance>" → "On-Prem" or "SaaS"

How to get an API token?

Access the Instana user-interface and  navigate to "Settings":

Under "Access Control", select  "API tokens":

Select a random token (which has all permissions granted). If none exists, simply create an API Token for use with the script:

Copy the API Token value and insert this in the <API_token>:


JSON-Dump file

The script exports JSON-files (<trace_Id>.json) into the current working directory, if trace-, parent- or spanId is missing!




Complete Trace (meaning a valid TraceId with Parent- and SpanId)

INFO: <traceId> OKINFO: e497c60f7d6b01d1 OK

API rate limit exceeded for 1 Token


Because of this limitation, we're using more than 1 token, to process more than 5000 traces per run!

But this means, there must be 1 token per 5000 calls → e.g.: You want to process 50.000 Calls, so you must have 10 API tokens!

INFO: API rate limit for zone default exceeded for API Token <API_token>
Rate Limit resets at <timestamp_for_token_reset>
Continue with new Token -> <new_API_token>

INFO: API rate limit for zone default exceeded for API Token OBV-yILYsYZLxisA
Rate Limit resets at 2019-04-09 15:43:09
Continue with new Token → TILuRjcFf9e9BoBv

ParentId is missingWARNING: Parent (SQL) Span missing in <traceId>WARNING: Parent (SQL) Span missing in e497c60f7d6b01d1
TraceId is missingWARNING: Trace <traceId> missingWARNING: Trace e411160f700a01d1 missing
SpanId is missing → The information sent by IronTap ("SQL related IFCIDs")ERROR: Missing z/IRIS data in <traceId>ERROR: Missing z/IRIS data in e497c60f7d6b01d1
SpanId found but label "SQL related IFCIDs" is missingWARNING: Expected label 'SQL related IFCIDs' in <traceId> not foundWARNING: Expected label 'SQL related IFCIDs' in e497c60f7d6b01d1 not found
Using an API token as an argument at the beginningERROR: API rate limit for Token <API_token> exceeded. Use another API token!ERROR: API rate limit for Token TILuRjcFf9e9BoBv exceeded. Use another API token!
There is already an API token with exceeded rate limitINFO: API Token with exceeded Limits: <API_token>INFO: API Token with exceeded Limits: OBV-yILYsYZLxisA
Instana Trace Summary

SUMMARY: Checked <all_traces> Traces 
<complete_traces> Complete Traces
<traces_without_traceId> TraceId missing 
<traces_without_parentId> ParentId missing 
<traces_without_spanId> SpanId missing

SUMMARY: Checked 5185 Traces 
144 Complete Traces 
5039 TraceId missing 
1 ParentId missing 
1 SpanId missing

Connection abort during runYou will get a Python RequestException and the script perform a shutdown('Connection aborted.', ConnectionAbortedError(10053, 'Eine bestehende Verbindung wurde softwaregesteuert\r\ndurch den Hostcomputer abgebrochen', None, 10053, None))
Filenames do not match the structure (caused by giving a false path as an argument)ERROR: File does not match! → <filename>ERROR: File does not match! → pyvenv.cfg
Directory does not exist (caused by giving a false path as an argument)ERROR: Directory <dir_name> does not exist!ERROR: Directory <json> does not exist!

  • No labels