This document will help users who wish to install, configure, and use z/IRIS v1.6.0. The following will be covered in this document:
- Install and customize z/IRIS IronTap (Linux server).
- Install and customize z/IRIS z/OS clients.
z/IRIS is a Java-based software product that offloads z/OS SMF to the Linux server, where the SMF records are translated, formatted and normalized into mainframe performance data. Thereafter, the mainframe performance data is transformed into traces valid for processing in application performance monitoring (APM) software platforms.
In summary z/IRIS was built to satisfy the following needs:
- Provide meaningful and readable z/OS Application and System Information for easy integration in to other distributed solutions and platforms.
- Integrate mainframe applications and systems monitoring with any APM platform to provide flexible end-to-end options for mainframe customers.
- Offload mainframe application and system information (SMF records) to ensure low impact on z/OS.
- Java-based application for zIIP eligibility on mainframe for improved performance and avoid MSU impact.
Downloading z/IRIS IronTap and z/OS Client installation packages
z/IRIS is owned, licensed and distributed by mainstorconcept GmbH. For more information about obtaining the required licenses to run z/IRIS, please email mainstorconcept at: firstname.lastname@example.org.
z/IRIS IronTap is a Java application server that runs on Linux and processes SMF records sent from z/IRIS clients. SMF records are transformed into trace records that can be processed in partner APM solutions.
All prerequisites and requirements are documented on the minimum hardware and software requirements page.
Any 3rd party solution that will receive and process data from z/IRIS must be active or running before z/IRIS IronTap is started.
Installing z/IRIS IronTap
- Create the IRONTAP_BASE_DIR folder '/opt/mainstorconcept/'.
- Extract the z/IRIS IronTap release zip file into the IRONTAP_BASE_DIR path.
- The z/IRIS IronTap files are located in the IRONTAP_INSTALL_DIR '/opt/mainstorconcept/irontap/'.
File Descriptions and Functions
contains z/IRIS IronTap modules and dependencies.
defines settings for the application. More info in "Configuring z/IRIS IronTap".
describes a launcher script for z/IRIS IronTap.
Configuring z/IRIS IronTap
The application.conf configuration file can be used to overwrite the default z/IRIS IronTap module configurations.
|global.time-zone||The default time zone for all z/OS systems.||CET|
|smf.ebcdic-codepage||The default EBCDIC encoding.||IBM1047|
|systems||An array of system ids and time zones.|
Multi system time zone config example
|worker-threads||The number of threads for parallel SMF processing.||-1||Choose the number of threads. A negative value will always result in using #cores -1 threads (at least 1).|
|logSmfDataOnError||An SMF parsing error will be logged including the whole SMF record.||true|
|port||The listener port which the z/IRIS client uses to connect to z/IRIS IronTap.|
|topicPattern||A pattern describing the topics z/IRIS IronTap should consume.||smf|
|consumer||Contains the Apache Kafka Consumer configuration provided by your Apache Kafka administrator.||default consumer config|
|bucket||The InfluxDB bucket to be used for z/IRIS metrics|
|org||The InfluxDB organization|
|url||The url to your InfluxDB|
|authenticateToken||A token with write permissions within the organization||Please note that all configs can be provided via the command line as well.|
Example config, overriding the default EBCDIC code page with IBM01140 and enabling Instana Events for mainframe jobs.
z/IRIS z/OS Client
z/IRIS clients are IBM JZOS based applications that stream SMF records to the Linux Server for processing and integration into 3rd party APM tools (z/IRIS IronTap).
Please refer to the minimum hardware and software requirements page.
IBM Java for z/OS
z/IRIS clients require IBM Java SDK 1.8. For more information on how to install and configure IBM Java for z/OS, please visit the IBM Knowledge Center.
z/IRIS clients stream SMF records from IBM SMF in-memory resources. For information on how to configure new SMF in-memory resources, please visit the IBM Knowledge Center.
Apache Kafka integration
z/IRIS clients activate Kafka Producers to stream SMF to a predefined and network accessible Kafka cluster. Users must download and install the Apache Kafka binaries in z/OS USS before configuring z/IRIS clients for Kafka streaming mode.
Visit the official Apache Kafka website to download binaries and access all relevant documentation about Apache Kafka Producers and Apache Kafka Clusters and more.
For more information about z/IRIS integration with Apache Kafka, visit Apache Kafka - Configuration Guide.
All z/IRIS clients must be granted access to SMF in-memory resources using SAF permissions. The RACFJCL text file is a RACF batch job to create a surrogate userid for the z/IRIS client started task with the required permissions for OMVS, SMF in-memory resources and MVS display commands.
Upload, customize and submit RACFJCL from the zIRISzOSClient.v%.%.%.zip installation package.
The settings detailed in the RACFJCL are not final, and some settings can be omitted. To ensure a problem free installation, consult your mainstorconcept Technical Professional regarding any changes to RACF permission in RACFJCL before proceeding with the installation.
ZFS directory ($APP_HOME)
Customize and run the following JCL example to create and mount a new ZFS directory for z/IRIS in USS. This directory is hereinafter referred to as $APP_HOME.
Install and configure a z/IRIS client
- Extract and upload (in binary mode) the zirisv%%.pax to the $APP_HOME directory on z/OS USS.
Unpack the file using the tar command:
Use the -v option only if you want to see verbose output.
Apply the authorized extended attribute to $APP_HOME/bin/CONSL001:
Make all commands in $APP_HOME/bin executable:
The z/IRIS license file is provided separately and must be uploaded to the $APP_HOME directory.
Configure the $APP_HOME/lib/ziris.comm.config to as required.
Parameters and options
Apache Kafka Producer properties when mode=kafka
The installation package contains default Apache Kafka Producer properties for z/IRIS streaming:
Avoid making changes to the default-ziris-producer.properties as this may be overwritten by updates.
Instead create a new producer.properties file by :
- Copy the sample producer properties file in the $KAFKA_HOME/config directory.
- Edit the properties as required.
- Specify the new producer properties path and file name in the ziris.comm.config file using the kafkaConfig parameter.
For more information about required properties for z/IRIS SMF streaming, see Apache Kafka - Configuration Guide .
Workload manager considerations
The z/IRIS started tasks should be allocated to low/discretionary priority in WLM.
Please consult your mainstorconcept technical professional if you require any assistance with amending your WLM policies.
Using z/IRIS clients
Create a z/IRIS client PROCLIB member
Upload the ZIRISPRC started task procedure and the ZIRISJVM JZOS procedure files from zIRISzOSClient.v%.%.%.zip to a concatenated PROCLIB data set.
- Change $APP_HOME in JCL to the $APP_HOME directory created for z/IRIS in ZFS.
If mode=kafka for SMF records streaming via a predefined and network accessible Kafka cluster, uncomment the following line and define the Kafka home directory to z/IRIS z/OS:
- Change $JVAHOME to the $JAVA_HOME directory where IBM Java version 8 resides.
Save the PROCLIB member.
Allocate the surrogate user id defined by the RACFJCL job:
Activating z/IRIS clients
To start the z/IRIS client Started Task, issue the following MVS Command:
where ziris is the member name of the z/IRIS PROCLIB member.
When z/IRIS reports a failed connection during start-up, check that either the z/IRIS Linux server is active or that the Apache Kafka cluster is running with the accessible topics and brokers.
Stopping z/IRIS clients
To stop the started task issue the following MVS Command:
Where ziris is the name of running z/IRIS STC Address Space.
z/IRIS client modify commands
z/OS users with SDSF and MVS Command Permissions, can interact with the z/IRIS client Started Task through MVS Modify Commands.
z/IRIS MVS Command Syntax :
ziris: is the name of the z/IRIS STC
cmd: is a valid z/IRIS Modify Command
lists and displays information about available commands:
displays a summary of current connections with SMF and the z/IRIS IronTap Linux Server.
z/IRIS client logging and error messages
- STDOUT and STDERR are merged by default into SYSOUT.
- The Started Task will send messages to the SYSLOG when the messages require immediate attention, for example a TCP connection was lost and will try to reconnect as defined in the ziris.comm.config file.
Example SYSOUT from z/IRIS client Started Task:
The Cloud Native Computing project OpenTelemetry provides an open standard for telemetry data, including traces and metrics. The importance and the capabilities of this open standard are recognized by countless APM vendors and ensure z/IRIS users have mainframe-inclusive tracing with proprietary and open-source APM tools. Because Observability and Monitoring requirements can vary greatly depending on the business case, implementing a uniform standard allows different APM products to interact with each other, resulting in custom solutions that meet our customer needs.
Read more about out OpenTelemetry support in our public knowledge base.
z/OS Connect trace streaming
z/IRIS creates OTLP compliant traces from z/OS Connect SMF type 123 version 2 records. DevOps teams have access to multiple performance indicators regarding REST API requests processed by z/OS Connect servers on mainframe, such as:
- the total elapsed time it took to send a response from z/OS.
- differentiation between z/OS connect transaction times and System-Of-Record(SOR) transaction times.
- mainframe related request and system identifiers for each REST API call that can be provided to SMEs for escalated root cause analysis.
- security authentication data.
- DevOps friendly trace tags that reduce the learning curve drastically when monitoring mainframe related application performance, e.g.:
See z/IRIS z/OS Connect support and configuration for examples of monitoring mainframe hosted Rest API calls in Jaeger with OpenTelemetry.
Load Balancing & High Availability with Apache Kafka Integration
Streaming SMF records through an intermediate predefined Kafka Cluster is now possible. Integration with Apache Kafka guarantees high availability and load balancing.
Using an intermediate Kafka cluster provides the ability to retain SMF records in the cluster for a configurable period and integrate the cluster into SMF record collection and eliminate traditional collection of SMF records using daily batch jobs on the mainframe.
The configuration options and recommendations for z/IRIS processing can be found here:
z/OS work activity tracing
Common address space work activity SMF records (Type 30) provide the following information:
- a batch job, APPC/MVS transaction program, OMVS forked or spawned address space starts
- a batch job or step, APPC/MVS transaction program, OMVS forked or spawned address space ends
- interval accounting for active jobs, started tasks, and system address spaces
- TSO session starts or ends (user logon and logoff)
The following performance information is provided by SMF 30 records:
- Work and batch job names and identifiers
- CPU and specialty engine utilization(Service units and CPU time)
- work start and end timestamps as well as duration/elapsed time
- Completion codes which signal whether an error occurred during processing
- Job Owners and assigned user ids
Benefits of streaming SMF 30 records in real-time:
- z/OS work monitoring on modern APM platforms
- Near real-time source for mainframe performance data which will increase the business value of existing mainframe capable infrastructure monitoring tools
- Near real-time mainframe resource consumption charge back calculations possibilities
- Incorporate mainframe administrators, developers and teams with modern DevOps and SysOps tools and processes
z/IRIS metric streaming
z/IRIS is capable of streaming metrics for storing and real-time processing in DBMS systems that are data sources for metric visualization and monitoring software.
Supported IBM SMF records types for z/IRIS metrics
70 subtype 1 - CPU activity metrics. See z/IRIS RMF Metrics Streaming for more information
- 123 subtype 1 version 1 and 2. See z/IRIS z/OS Connect Metrics Streaming for more information
Additional SMF records coming soon but can be explicitly requested.
Officially supported metric data sinks
Support for alternate data sinks can be requested.
z/IRIS metrics comply to open standards so that customers are not limited by proprietary solutions.
Distributed Db2 for z/OS tracing streaming
Db2 deadlock traces
Deadlock traces provide details regarding deadlock events that have occurred within a Db2 address space. z/IRIS processes and provides the application Holder and Waiter information from a deadlock event as trace-ids that are searchable within partner APM solution. This helps DevOps teams quickly locate applications holding Db2 resources as well as identify the applications who timed out due to the unavailability of these Db2 resources.
Db2 accounting traces
Accounting information gives an overview of a unit-of-work completed for JDBC client application. Details include elapsed CPU seconds and response time, SQL requests summaries, amount of fetched data and commits or rollbacks.
Accounting trace details vary according to the classes applied when activating accounting tracing. Speak to your Db2 administrator to ensure your APM integration contains all the information your DevOps teams require for effective monitoring.
To receive records for each Unit-Of-Work in the Db2 Address Space, the DB2 ZPARM ACCUMACC must be set to NO. If ACCUMACC is left active, the accounting records are rolled-up by the chosen aggregation field also specified in the ZPARM. Please see the IBM web pages for more information regarding the affected parameters:
DDF/RRSAF ACCUM field (ACCUMACC subsystem parameter)
AGGREGATION FIELDS field (ACCUMUID subsystem parameter)
Db2 performance traces
This Db2 performance traces provide low-level performance information. Activating these traces can affect Db2 for z/OS response time and CPU consumption, and should only be activated when required.
IBM does not recommend that users run this performance class traces in a production environment over extended periods of time.