Page tree
Skip to end of metadata
Go to start of metadata

Goals

  • Bypass known Instana UI search limitations, specifically the Analyze view, by using the REST API to ensure all spans delivered by z/IRIS are available in the Instana Backend
  • Detailed analysis of the delivered spans
    • Are the span identifiers (traceId, parentId, spanId) available in the back-end
    • If not, which identifiers could not be found.

Background and strategic fit

Some traces are invisible in the Instana UI, although the z/IRIS IronTap component reports a successful upload of all valid Instana  spans.

This script intends to clarify how many mainframe related traces cannot be located and provides information about the missing traces or calls: traceId, parentId or spanId.

This script was written in Python.

Requirements

  1. A linux system to execute the script

  2. IronTap dump files with the subsequent structure:"YYYY-MM-DDTHH.MM.SS.NNN+HH.MM-HttpResponseCode.json" (e.g.: 2019-04-11T10.40.38.573+02.00-204.json)
    Example of file content:


    JSON
    [  
       {  
          "spanId":"7d52ec37524f079d",
          "traceId":"55248b6b0ad555c7",
          "timestamp":1554972044358,
          "duration":0,
          "name":"SQL related IFCIDs",
          "type":"ENTRY",
          "data":{  
             "IFCIDs":[  
                {  
                   "Aggregated smf overview":[  
                      {  
                         "Total records":1
                      },
                      {  
                         "IFCID 3 - Accounting Data Collection":[  
                            {  
                               "Instrumentation Data":[  
                                  "Acctg. Start Time:2019-04-11T08:40:44.341407",
                                  "Acctg. End Time:2019-04-11T08:40:44.358031",
                                  "Response Time:16624 micros",
                                  "CPU Start Time:26990 micros",
                                  "CPU End Time:43464 micros",
                                  "Acctg. Reason:DDF Type 2 Becoming Inactive",
                                  "Network Id:",
                                  "2Phase Commit Requests:1",
                                  "Abort Requests:0",
                                  "SU Conversion Factor:12497",
                                  "Workload Manager Service Class Name:DB2SSYST"
                               ]
                            },
                            {  
                               "DDF Data":[  
                                  "Threads to roll into QLAC Section:1",
                                  "Remote Location Name:::192.168.10.53",
                                  "Remote Product ID:JCC04210",
                                  "Msgs Sent to Remote Site:3",
                                  "Msgs Recv. from Remote Site:3",
                                  "SQL Stmts Recv. from Remote Site:3",
                                  "Bytes Sent to Remote Site:578",
                                  "Bytes Recv. from Remote Site:456",
                                  "Rows Sent to Remote Site:1",
                                  "Block Fetch Blocks Transmitted:2",
                                  "Single/Two Phase Commits Recv.:1"
                               ]
                            },
                            {  
                               "Lock Usage Data":[  
                                  "Resource Limit Id:",
                                  "Lock Requests:3",
                                  "Unlock Requests:1",
                                  "Claim Requests:2"
                               ]
                            },
                            {  
                               "MVS & DDF  Data":[  
                                  "Product Version:04",
                                  "Product Release:21",
                                  "Product Modification Level:0",
                                  "JCC Client Platform:192.168.10.53",
                                  "JCC Client Appl. Name:db2jcc_application",
                                  "JCC Client Auth Id:insusr1",
                                  "JCC Account Suffix:JCC04210192.168.10.53"
                               ]
                            },
                            {  
                               "SQL Data":[  
                                  "Prepares:1",
                                  "Opens:1",
                                  "Byte Flag:",
                                  "Cache Copied Prepared Stmt Reqs.:1",
                                  "Rows Fetched:1"
                               ]
                            },
                            {  
                               "Buffer Manager Data":[  
                                  "Get Page Requests:34",
                                  "Dynamic Prefetch Requested:3"
                               ]
                            }
                         ]
                      }
                   ]
                }
             ],
             "tags":{  
                "correlation":{  
                   "luwid":"C3F0C1F8F0C1F3F5C4F0C2C640404040D5F4CCC370AD00EA",
                   "ace":483061536,
                   "networkId":"C0A80A35",
                   "luName":"D0BF    ",
                   "uniquenessValue":235247384096941,
                   "commitCount":234
                },
                "ifcid":{  
                   "3":true
                },
                "class":"accounting",
                "database":{  
                   "instance":"DALLASC         ",
                   "version":12,
                   "type":"sql",
                   "connection":{  
                      "jcc":{  
                         "applicationName":"db2jcc_application  ",
                         "authId":"insusr1 "
                      }
                   }
                },
                "db2zos":{  
                   "records":1,
                   "time":{  
                      "cpu":{  
    
                      },
                      "elapsed":{  
                         "wait":{  
    
                         },
                         "numberOf":{  
    
                         }
                      }
                   },
                   "stmtType":{  
    
                   },
                   "numberOf":{  
                      "sqlStmtsReceived":3,
                      "locks":{  
                         "requests":{  
                            "lock":3,
                            "unlock":1,
                            "claim":2
                         }
                      },
                      "stmtTypes":{  
                         "prepare":1,
                         "open":1
                      }
                   }
                },
                "accounting":{  
                   "reason":"0000000B"
                }
             }
          },
          "parentId":"f8480c1f9d5aef9b"
       }
    ]
    

User interaction and design

The script uses the Instana REST API to compare the traces from the file with those displayed in the Instana Analyze View.

API Calls used:

  • Trace Detail: {basic_instana_url}/api/application-monitoring/analyze/traces;id={id} (for On-Prem instance v1.147.536 or earlier) or  {basic_instana_url}/api/application-monitoring/analyze/traces/{id} (for SaaS instance v1.151.124 or earlier)→ gives us a JSON with all information about this trace and its child spans

How does the Script work?

A zip-file called: InstanaTraceSummary.zip is provided


Choose a directory as the location to unzip the .zip file content.

Script name: InstanaTraceSummary.py

Requirements

Before executing the script, please verify:

  1. Is Python installed? → If not, execute: apt-get install python3.6 (or later)
  2. Is pip installed? → If not, execute: apt-get install python3-pip
  3. If both is available, check if the module "requests" is installed (you can check this by going into the python prompt and use the command help("modules") to get a list of all installed modules or help("<module_name>") to search for a specific module) → If not, execute: python3.6 -m pip install requests

Once the above prerequisites have been met, execute the script.

The subsequent syntax ensures a successful execution of the script:

COMMAND
python3.6 "<path_to_InstanaTraceSummary.py-file" -p "<path_to_IronTap_json_folder>" -b "<basic_instana_url>" -t '<API_token>' -i "<instana_instance>"


IMPORTANT

The single quotation marks are mandatory: '<API_token>' .

Example:

python3.6 InstanaTraceSummary.py -p "/opt/json" -b "http://instana01.xxx.com" -t '12345LKZWOOPLK' -i "on-prem"


Arguments

  • -p "<path_to_IronTap_json_folder>" → this has to be an absolute path to the "json"-folder where the debug files are present.
  • -b "<basic_instana_url>" → enter the URL to your instana on-prem or SaaS instance
  • -t '<API_token>' → any API token to authorize REST API calls
  • -i "<instana_instance>" → "On-Prem" or "SaaS"

How to get an API token?

Access the Instana user-interface and  navigate to "Settings":


Under "Access Control", select  "API tokens":


Select a random token (which has all permissions granted). If none exists, simply create an API Token for use with the script:

Copy the API Token value and insert this in the <API_token>:



Output

JSON-Dump file

The script exports JSON-files (<trace_Id>.json) into the current working directory, if trace-, parent- or spanId is missing!



Case

Output

Example

Complete Trace (meaning a valid TraceId with Parent- and SpanId)

INFO: <traceId> OKINFO: e497c60f7d6b01d1 OK

API rate limit exceeded for 1 Token

INFORMATION:

https://instana.github.io/openapi/#section/Backend-REST-API

Because of this limitation, we're using more than 1 token, to process more than 5000 traces per run!

But this means, there must be 1 token per 5000 calls → e.g.: You want to process 50.000 Calls, so you must have 10 API tokens!

INFO: API rate limit for zone default exceeded for API Token <API_token>
LIMIT: 5000, REMAINING CALLS: 0
Rate Limit resets at <timestamp_for_token_reset>
Continue with new Token -> <new_API_token>

INFO: API rate limit for zone default exceeded for API Token OBV-yILYsYZLxisA
LIMIT: 5000, REMAINING CALLS: 0
Rate Limit resets at 2019-04-09 15:43:09
Continue with new Token → TILuRjcFf9e9BoBv

ParentId is missingWARNING: Parent (SQL) Span missing in <traceId>WARNING: Parent (SQL) Span missing in e497c60f7d6b01d1
TraceId is missingWARNING: Trace <traceId> missingWARNING: Trace e411160f700a01d1 missing
SpanId is missing → The information sent by IronTap ("SQL related IFCIDs")ERROR: Missing z/IRIS data in <traceId>ERROR: Missing z/IRIS data in e497c60f7d6b01d1
SpanId found but label "SQL related IFCIDs" is missingWARNING: Expected label 'SQL related IFCIDs' in <traceId> not foundWARNING: Expected label 'SQL related IFCIDs' in e497c60f7d6b01d1 not found
Using an API token as an argument at the beginningERROR: API rate limit for Token <API_token> exceeded. Use another API token!ERROR: API rate limit for Token TILuRjcFf9e9BoBv exceeded. Use another API token!
There is already an API token with exceeded rate limitINFO: API Token with exceeded Limits: <API_token>INFO: API Token with exceeded Limits: OBV-yILYsYZLxisA
Instana Trace Summary

----------------------------------------------
SUMMARY: Checked <all_traces> Traces 
<complete_traces> Complete Traces
<traces_without_traceId> TraceId missing 
<traces_without_parentId> ParentId missing 
<traces_without_spanId> SpanId missing

----------------------------------------------
SUMMARY: Checked 5185 Traces 
144 Complete Traces 
5039 TraceId missing 
1 ParentId missing 
1 SpanId missing

Connection abort during runYou will get a Python RequestException and the script perform a shutdown('Connection aborted.', ConnectionAbortedError(10053, 'Eine bestehende Verbindung wurde softwaregesteuert\r\ndurch den Hostcomputer abgebrochen', None, 10053, None))
Filenames do not match the structure (caused by giving a false path as an argument)ERROR: File does not match! → <filename>ERROR: File does not match! → pyvenv.cfg
Directory does not exist (caused by giving a false path as an argument)ERROR: Directory <dir_name> does not exist!ERROR: Directory <json> does not exist!



  • No labels