This document will help users who wish to install, configure, and use z/IRIS.
Purpose of this document
This document will assist users who wish to:
Understand the requirements for a z/IRIS installation
Install and customize z/IRIS IronTap on Linux
Install and customize z/IRIS z/OS Client
Troubleshooting z/IRIS z/OS Client
z/IRIS is a Java based software solution which offloads z/OS SMF records, translates and formats the SMF records into traces and integrates the traces into other systems, applications or products which can process this information.
z/IRIS is a Client-Server solution in Java and High Level Assembler ( HLASM ). The z/OS Client ( IBM Java 8 ) connects to the z/IRIS IronTap Linux server, which processes, parses and normalizes the SMF records from z/OS into traces and forwards these traces to 3rd party applications or products.
In summary z/IRIS was built to satisfy the following needs:
Provide meaningful and readable z/OS Application and System Information for easy integration in to other distributed solutions and platforms.
Open-up the mainframe to cutting-edge technologies without service providers requiring in-depth understanding of the mainframe or z/OS to cater for mainframe customers and environments.
Offload mainframe application and system information for processing outside mainframe using Java to reduce processing impact on the mainframe.
Give the mainframe the flexibility to be included in modern performance monitoring platforms otherwise segregated from mainframe architecture and infrastructure.
z/IRIS IronTap is a software product written in Java, which parses detailed information from the z/OS client and transforms it in to an appropriate format for further processing.
Using a streamlined connection directly to the z/IRIS z/OS Client IronTap processes live z/OS trace data, near real-time. For a list of available trace features with z/IRIS, see z/IRIS Trace Features
maxSmfToBeProcessed: Default=500000 maximum amount of SMF records kept in memory
If this number is exceeded, all queued SMF records will be discarded.
maxSmfPerDB2Thread: Default=300000 maximum amount of SMF records kept in memory per Db2 jdbc thread
If this number is exceeded, all queued SMF records associated with a Db2 JDBC thread will be discarded.
logSmfDataOnError: Default=true A SMF parsing error will be logged including the whole SMF record
If you don't want to dump the SMF records, you can disable this feature by setting it to 'false'.
Using z/IRIS IronTap
Navigate to the IRONTAP_INSTALL_DIR
Run the following command to start the application: ./run.sh
Use the following command to see the log information and statistics in real time:
tail -f ./logs/irontapInfo.log
Below is an example of the expected output:
The following message indicates that a z/IRIS z/OS Client is connected:
z/IRIS z/OS Client connected
New smf server connected: <ipAddress>"
The following message indicates that the span dump path has been created, based on the instanaSpanDumpDirectoryPath value in the config.yamlfile:
Writing Instana spans to <path to directory>
The following example is used to explain the z/IRIS IronTap logging:
Active connections: 1 The number of z/IRIS z/OS clients currently connected to the IronTap server (max. 128 connections)
SMF record count: 136 (57 KByte) This shows the total number of records as well as the amount of data the system received since its startup
SMF queue: 0 Number of records in the input queue
to be processed: 0 Number of records for which the current transaction is not finished yet
discarded: 76/136 Number of records which won't be used for tracing. (In the above example: There are 136 records in total of which 76 have been discarded)
Match count: 6 Number of spans resulted from performance record correlation.
This number is only visible when performance records are enabled. Accounting records are being sent directly to the agent, while the performance records depend on a particular order
Enqueued for dispatch: 30/30 spans Number of spans ready to send to Instana
dropped: 0 Number of spans dropped
sent to agent: 30 For Instana APM integration: Number of spans actually sent to the Instana agent (one span can contain multiple SMF records)
Activating debug features can have a performance impact on the system
This feature is enabled by setting a non empty string for the instanaSpanDumpDirectoryPath property. IronTap will write all spans into the provided directory. Multiple spans may be joined together into a JSON array in one file.
Each file name consists of a timestamp (when this span was sent to the agent) followed by a dash and the agent HTTP response code or the string "connection-error" if no connection could be established.
This feature is enabled by setting a non empty string for the smfDumpFilePath property. Irontap will write all SMF records to disk. You may provide either a filename or a directory. If you provide a directory IronTap will create a new file during each system start e.g.: 2019-01-24T14.30.00.00.smf. If you provide a filename IronTap will always append to the file.
During high SMF throughput from the z/IRIS z/OS Client, SMF dump files can become very large in a short amount of time.
z/IRIS Irontap Integration Considerations
You must have a running Instana agent otherwise IronTap is not able to send Spans to the Instana APM back-end.
In some cases end-to-end application tracing into the mainframe requires a trace tag to be inserted on the distributed calls, for example Distributed Db2 for z/OS tracing. To configure end-to-end tracing, the required instrumentation must be enabled by setting the appropriate JVM options.
z/IRIS z/OS Client
The z/IRIS z/OS Client is a IBM JZOS based application, built to connect to the z/IRIS Linux server (IronTap) and forward selected z/OS application and system information.
Setting up the z/IRIS z/OS Client is simple and but will require users with access to:
Updating SMFPRMxx Parmlib member
Add and Change RACF User and Resource definitions
Creating and Mounting zFS directories
Applying extended attributes to OMVS files
Modifying OMVS file attributes
Once all configuration has been completed, the z/IRIS z/OS Client is activated through MVS as a started task and runs in its own address space until a shutdown command is issued, using standard MVS commands.
All z/IRIS software, including z/IRIS z/OS Client can be downloaded from the mainstorconcept Download Portal.
New users should register here first, before attempting to download any software.
All required z/OS licenses are obtained through a valid license agreement. For more information please email us at: email@example.com
Preparing your mainframe environment for the z/IRIS z/OS Client
The following is assumed from this point:
The user intending to install and configure z/IRIS has the necessary authority to do so.
The required z/IRIS IronTap Server is configured and available for connection
z/IRIS IronTap and z/OS Client are licensed property of mainstorconcept GmbH, the use of this product is therefore subject to a license agreement which provides access to license key files required to make use this software.
z/OS version 2.1 or later is installed and configured appropriately
Java preparations for z/OS
Download IBM SDK for zOS component
At the time of writing this document, IBM Java SDK 1.8 was the most recent package available for z/OS. z/IRIS z/OS Client was built using the Java SE 8 capability only. Obtain the Java SDK from IBM.
All required documentation should be available at IBM.
Security preparations for z/OS with RACF
SMF In-Memory Resources can only be accessed once they have been defined to RACF. Decide on an In-Memory Resource Name for DB2 SMF Record types 102, with a maximum of 26 characters. Example IFASMF.INMEM.SMF102 The RACF Resource Name must be the same as the name defined above, prefixed with "IFA." as a prefix. Therefore, the RACF resource name for the above In-Memory Resource is: IFA.IFASMF.INMEM.SMF102. Define the Resource to RACF class Facility, using the following TSO Command:
The zIRISzOSClient.v%.%.%.zip contains a file called RACFJCL.txt. This is a z/OS Job which creates a surrogate userid for z/IRIS using IBM RACF commands. Upload this file via FTP as text, customize the job as required and submit.
The settings detailed in the RACFJCL are not final, and some settings can be omitted. To ensure a problem free installation, consult you mainstorconcept Technical Professional regarding any changes to RACF permission in RACFJCL before proceeding with the installation.
For customers not using IBM RACF as a Security Server on z/OS, the appropriate access must be provided to the userid associated with the started task as well as for the In-Memory Resource. For assistance, please contact mainstorconcept GmbH.
SMF preparations for the z/IRIS z/OS Client
All in-memory resource must follow the naming standard: IFASMF.rname
See Defining in-memory resources at the IBM KnowledgeCenter for further details.The z/IRIS z/OS client automatically connects to any resource with prefixed IFASMF.INSTANA.*
Any resource name may be used.
If the second qualifier is not Instana, the resource will be defined in the ziris.comm.config $APP_HOME/lib file later on in the configuration.
This document only describes the required steps for the purposes of the z/IRIS application, should further information be required please refer to the IBM Documentation. Create a new SMFPRMxx member in the appropriate concatenated PARMLIB data set. Copy all settings from the active SMFPRMxx member and save the new member. Add the following SMFPRMxx parameters:
$INMEM_NAME is the In-Memory Resource Name for DB2 SMF Record types 102 from chapter 3.2
RESSIZMAX(nM) defines the resource maximum size, in this case 128M. Up to 2GB can be defined if required.
When the resource reaches the maximum defined here,the in-memory resource wraps and overwrites the oldest SMF records with the newest SMF records.
mainstorconcept GmbH may have provided you with recommended size which should be used here, otherwise System Programmers and Administrators have discretion on this parameter, based on the rate at which SMF Records will be generated.
TYPE(ttt) defines the SMF Record Type that is written to the in-memory resource. The type is dependent on the Features which your organisation wants to take advantage of.
The following is an example SMFPRMxx member. Here we see the In-Memory Resource Name described in chapter 3.2, SMF Record Types 102 will be written to this in-memory resource, and the maximum size is 128 MB.
In this example the Logstream includes SMF Type 102 as well, which means these records will be written to both the Logstream and the in-memory resource. To avoid this duplication, exclude the SMF type 102 from the Logstream definition explicitly.
Save the PARMLIB member
At this point, please ensure all Db2 traces have been stopped. To do so, issue the following Db2 command:
Issue the following MVS command to activate the new In-Memory resource dynamically:
Where xx is equal to the suffix of the new SMFPRMxx parmlib you created in this chapter. The log output should reflect the new In-Memory Resource, similar to the following output:
Change the IEASYSxx PARMLIB member to point to this new SMFPRMxx member.
Create a new ZFS directory, this directory will be referred to as $APP_HOME in this document. You may want to create a new mount entry for this zFS directory depending on your sites requirements. The minimum amount of space required is for z/IRIS: 2000K
Install and Configure z/IRIS
Upload and Install z/IRIS
In the zIRISzOSClient.v%.%.%.zip is a tar file called zirisv100.pax
Upload this file to the $APP_HOME directory created in STEP 3.4 in binary mode .
Once the file has been uploaded, unpack the file using the tar command:
pax -w[v]f zirisv100.pax
Use the -v option only if you want to see verbose output.
Take not of the extended attributes associated with some of the files. If the files do not match this criteria, you will have to apply the extended attributes manually.
Edit the ziris.comm.config using the syntax and parameter information detailed below.
ssl=N[o] | Y[es]
alias=Alias_Name This is a locally defined alias name that refers to the remote z/IRIS IronTap Server Agent. This can be any name with alphanumeric characters, without spaces or special characters (i.e. @!§$%&/() etc.) It is highly recommended that the Alias name be kept short for readability.
mode=tcp|udp|mq Default:tcp This defines the desired network communication protocol, TCP or UDP. TCP is default and supported by Remote z/IRIS IronTap server. UDP and MQ are for future use only.
host=IPv4_address:Port_Number IPv4_Address: must be the IP address of the remote z/IRIS IronTap server Port_Number: must be the connectable port of the remote z/Iris Iron Tap Linux Server
timeout=timeout_value_seconds The amount of time in seconds used as a connection and communication timeout value. This can be any integer value, but must be within the norms of the sites network.
inmem=SMF_Inmem_Resource SMF_Inmem_Resource to the SMF in-memory resource defined in the SMFPRMxx parmlib member.
If the in-memory resource name is prefixed with IFASMF.INSTANA, the z/IRIS z/OS Client will automatically connect to it, and should not be specified here.
All in-memory resources that have been created for z/IRIS should be listed here.
Mutliple in-memory resources will be defined as follows:
ssl=N[o] | Y[es] Default:Yes Enable or disable SSL encrypted and trusted communication with the remote z/IRIS IronTap server
reconnect=999 Default: 10 Configure the amount of seconds z/Iris will wait before trying to reconnect to the remote z/IRIS IronTap server Any valid Integer representing seconds is accepted
maxTCPSend=99999 Default: 65536 Configure the maximum number of bytes allowed for TCP traffic from z/Iris to the remote z/IRIS IronTap server. This should be equal to or less the than z/OS TCP max send buffer size.
cicshlq=cicstshlq Default: None Enter the HLQ for CICS Transaction Server data sets. For example the HLQ for the CICS SDFHLOAD, data set
The extended attribute for the $APP_HOME/bin/CONSL001 must be: +a : apf authorized +s : shared address space issue the following command:
extattr +a ./CONSL001
All files within the $APP_Home/bin path must be executable, issue the following command:
chmod +x ./*
Workload Manager Considerations
The z/IRIS STC should be defined with low / discretionary priority in WLM. (i.e. not in the SYSSTC service class)
Please consult your mainstorconcept Technical Professional if you require any assistance in this regard.
Using the z/IRIS z/OS Client
The z/IRIS z/OS Client Started Task
Upload the ZIRIS.jcl file from the zIRISzOSClient.v%.%.%.zip to a concatened PROCLIB Dataset.
MNDUP Step in the zIRIS.jcl proclib member is currently optional. This has been included in preparation for further z/IRIS CICS TG/TS performance information streaming in future releases
Change $APPHOME in JCL to the $APP_HOME directory created for z/IRIS in zFS
Change $JVAHOME to the $JAVA_HOME directory where IBM Java version 8 resides
Save the job as ZIRIS. If ZIRIS is not available, you may use a different name, but please note that this job name is related to the job name in the RACFJCL for CLASS STARTED permissions. If the job name is not ZIRIS, you will need to amend the permissions for the surrogate user ID created:
RDEFINE STARTED JOBNAME.** UACC(NONE) STDATA(USER(ZIRIS) +
Upload the ZIRISJVM jcl file to a concatenated PROCLIB data set from the zIRISzOSClient_v100.zip file. This proclib member contains the required JVM options to start the z/IRIS z/OS Client
Starting the Started Task
To start the z/IRIS z/OS client started task, issue the following MVS Command:
where zirisis the member name of the ZIRIS proclib member
Should a failed connection message be displayed during initial connection with the z/IRIS IronTap Linux Server, please check that the z/IRIS Linux Server instance is active and ports are open for a connection.
Stopping the Started Task
To stop the started task issue the following MVS command:
Where zirisis the name of running z/IRIS STC address space.
z/IRIS z/OS Logging and Error Messages
STDOUT and STDERR are merged by default into SYSOUT.
The Started task will send messages to the SYSLOG when the messages require immediate attention, for example a TCP connection was lost and will try to reconnect as defined in the ziris.comm.config file.
Example SYSOUT from z/IRIS z/OS Client started task:
Interacting with z/IRIS through MVS modify commands
z/OS users with SDSF and MVS command permissions, can interact with the z/IRIS Client started task through MVS modify commands. z/IRIS MVS comsand syntax :
where: ziris: is the name of the z/IRIS STC cmd: is a valid z/IRIS modify command
z/IRIS MVS Commands
lists and displays information about available commands:
displays a summary of current connections with SMF and the z/IRIS IronTap Linux Server
RESPONSE=CPAC BPXM023I (SMFRAPI) ZIRIS will accept the STOP(P) and
RESPONSE=CPAC BPXM023I (SMFRAPI)
RESPONSE=CPAC ZIRIS STATUS Jul 18, 2018, 12:01:34 PM
RESPONSE=CPAC TCP Connection: active to /172.25.999.999:9999
RESPONSE=CPAC Number of Listeners: 1
RESPONSE=CPAC Listener 1 streaming from IFASMF.INMEM.SMFDB2
z/IRIS Features and Trace requirements for z/OS
All z/IRIS Features are built to extract specific information from the mainframe, which is parsed and normalized before being given over to a third party application. Each feature will have its own tracing requirement, dependent on the component being traced and the complexity of tracing the component.
Feature: Distributed Db2 for z/OS Tracing
With z/IRIS, mainframe customers and ISVs can now correlate distributed Db2 for z/OS calls made using JDBC 4.0 . From performance traces which delivers low-level performance information (but are very expensive) to accounting and deadlock events traces.
Deadlock event traces provide from mainframe help users connect Deadlocks and timeouts experience as well as see which application was holding a resource up during the deadlock scenario.
The Transaction-Trace-Set provides trace information per UOW (grouped by commit or rollback) provides information about the JDBC transaction: response and processor timers per transaction, types of SQL work performed, the amount of statements processed etc. Transaction-trace-sets are much less expensive than the performance trace sets.
This trace-set provides detailed low-level performance information regarding JDBC SQL calls processed Db2 Address space, detailing among other information:
To enable this trace-set for to be processed by z/IRIS, the following Db2 Trace command must be issued to the relevant Db2 Subsystem:
Your System programmer must ensure that the correct SMF in-memory resources have been defined to capture the SMF Record Types 102.
Refer to the SMF Preparations in this document for more information.
Use this trace set in pre-production or development environments, as part of QA
It is not recommended to run this type of trace in a productive environment.
This trace-set provides more in-depth Db2 information regarding a deadlock that occurred. We provides information regarding the Holder and Waiter information, which assist Db2 administrators and programmers track the affected applications as well as the initial holder during this deadlock scenario.
To enable this trace-set to be processed by z/IRIS, the following Db2 Trace command must be issued to the relevant Db2 Subsystem:
Your System programmer must ensure that the correct SMF in-memory resources have been defined to capture the SMF Record Types 102.
Refer to the SMF Preparations in this document for more information.
This trace-set captures the accounting information, grouped by Unit-of-Work, for JDBC calls processed at the Db2 Address space. The Unit-Of-Work is ended at commit or rollback by the client Db2 application.
To enable this trace-set to be processed by zIRIS, the following Db2 Trace command must be issued to the relevant Db2 Subsystem:
In order to receive records for actual UOWs in the Db2 Address space, the DB2 ZPARM ACCUMACC must be set to NO. if ACCUMACC is left active, the accounting records are rolled-up by the chosen aggregation field also specified in the ZPARM. Please see the IBM web pages for more information regarding the affected parameters:
DDF/RRSAF ACCUM field (ACCUMACC subsystem parameter)
AGGREGATION FIELDS field (ACCUMUID subsystem parameter)