Skip to end of metadata
Go to start of metadata

Summary

This document will help users who wish to install, configure, and use z/IRIS v1.4.x. The following will be covered in this document:

  • Install and customize z/IRIS IronTap (Linux server).
  • Install and customize z/IRIS z/OS clients.

Background

z/IRIS is a Java-based software product that offloads z/OS SMF to the Linux server, where the SMF records are translated, formatted and normalized into mainframe performance data. Thereafter, the mainframe performance data is transformed into traces valid for processing in application performance monitoring (APM) software platforms.

In summary z/IRIS was built to satisfy the following needs:

  • Provide meaningful and readable z/OS Application and System Information for easy integration in to other distributed solutions and platforms.
  • Integrate mainframe applications and systems monitoring with any APM platform to provide flexible end-to-end options for mainframe customers.
  • Offload mainframe application and system information (SMF records) to ensure low impact on z/OS.
  • Java-based application for zIIP eligibility on mainframe for improved performance and avoid MSU impact.

Downloading z/IRIS IronTap and z/OS Client installation packages

z/IRIS IronTap installation packages can be downloaded from the mainstorconcept Download Portal. New users should register here first, before accessing the portal.

z/IRIS is owned, licensed and distributed by mainstorconcept GmbH. For more information about obtaining the required licenses to run z/IRIS, please email mainstorconcept at: info@mainstorconcept.com.

z/IRIS IronTap

z/IRIS IronTap is a Java application server that runs on Linux and processes SMF records sent from z/IRIS clients. SMF records are transformed into trace records that can be processed in partner APM solutions.

Requirements 

All prerequisites and requirements are documented on the minimum hardware and software requirements page.

Any 3rd party solution that will receive and process data from z/IRIS must be active or running before z/IRIS IronTap is started.

Installing z/IRIS IronTap

  • Create the IRONTAP_BASE_DIR folder '/opt/mainstorconcept/'.
  • Extract the z/IRIS IronTap release zip file into the IRONTAP_BASE_DIR path.
  • The z/IRIS IronTap files are located in the IRONTAP_INSTALL_DIR '/opt/mainstorconcept/irontap/'.

File Descriptions and Functions

libs/

contains z/IRIS IronTap modules and dependencies.

application.conf

defines settings for the application. More info in "Configuring z/IRIS IronTap".

run.sh

describes a launcher script for z/IRIS IronTap.

Configuring z/IRIS IronTap 

The application.conf configuration file can be used to overwrite the default z/IRIS IronTap module configurations.

ModuleProperty
Default valueNotes
zos

global.time-zoneThe default time zone for all z/OS systems.CET

smf.ebcdic-codepageThe default EBCDIC encoding.IBM1047

systemsAn array of system ids and time zones.
 Multi system time zone config example
systems: [
{id: S0W1, time-zone: EST}
{id: S0W2, time-zone: CET}
]
core

worker-threadsThe number of threads for parallel SMF processing.-1Choose the number of threads. A negative value will always result in using #cores -1 threads (at least 1).

logSmfDataOnErrorAn SMF parsing error will be logged including the whole SMF record.true
sources.tcp

portThe listener port which the z/IRIS client uses to connect to z/IRIS IronTap.

sources.kafka

topicPatternA pattern describing the topics z/IRIS IronTap should consume.smf

consumerContains the Apache Kafka Consumer configuration provided by your Apache Kafka administrator.default consumer config
storage.mertrics.influxdb2

bucketThe InfluxDB bucket to be used for z/IRIS metrics


orgThe InfluxDB organization


urlThe url to your InfluxDB


authenticateTokenA token with write permissions within the organization
Please note that all configs can be provided via the command line as well.

 Example config
irontap{
 storage{
  metrics{
   influxdb2{
    bucket = "16b45434a5f58182"
    org = "e9043121e23a02da"
    url = "https://influx-cluster.com:32000"
    authenticateToken = "MySecureOrgToken=="
   }
  }
 }
}

Example config, overriding the default EBCDIC code page with IBM01140 and enabling Instana Events for mainframe jobs.

application.conf dot notation
irontap.zos.smf.ebcdic-codepage: "IBM01140"
irontap.apm.instana.sdsf.returnCodeEventsEnabled: true


application.conf Json notation
irontap{
 zos{
  smf{
   ebcdic-codepage: "IBM01140"
  }
 }
 apm {
  instana{
   sdsf{
    returnCodeEventsEnabled: true
   }
  }
 }
}


z/IRIS z/OS Client

z/IRIS clients are IBM JZOS based applications that stream SMF records to the Linux Server for processing and integration into 3rd party APM tools (z/IRIS IronTap).


 Click here to expand requirements...

Requirements 

Please refer to the minimum hardware and software requirements page.

IBM Java for z/OS

z/IRIS clients require IBM Java SDK 1.8. For more information on how to install and configure IBM Java for z/OS, please visit the IBM Knowledge Center.

In-Memory Resources

z/IRIS clients stream SMF records from IBM SMF in-memory resources. For information on how to configure new SMF in-memory resources, please visit the IBM Knowledge Center.

Apache Kafka integration

z/IRIS clients activate Kafka Producers to stream SMF to a predefined and network accessible Kafka cluster. Users must download and install the Apache Kafka binaries in z/OS USS before configuring z/IRIS clients for Kafka streaming mode.  

Visit the official Apache Kafka website to download binaries and access all relevant documentation about Apache Kafka Producers and Apache Kafka Clusters and more.

For more information about z/IRIS integration with Apache Kafka, visit Apache Kafka - Configuration Guide.



RACF configuration 

All z/IRIS clients must be granted access to SMF in-memory resources using SAF permissions. The RACFJCL text file is a RACF batch job to create a surrogate userid for the z/IRIS client started task  with the required permissions for OMVS, SMF in-memory resources and MVS display commands.

Upload, customize and submit RACFJCL from the zIRISzOSClient.v%.%.%.zip installation package.

The settings detailed in the RACFJCL are not final, and some settings can be omitted. To ensure a problem free installation, consult your mainstorconcept Technical Professional regarding any changes to RACF permission in RACFJCL before proceeding with the installation.


Customers who use alternate SAF software can contact mainstorconcept for assistance if required.


ZFS directory ($APP_HOME)

Customize and run the following JCL example to create and mount a new ZFS directory for z/IRIS in USS. This directory is hereinafter referred to as $APP_HOME.

Sample JCL to create z/IRIS zFS directory
// JOB
//* Create a z/IRIS ZFS directory
//CREATE EXEC PGM=BPXBATCH,
// PARM='SH mkdir -p /usr/lpp/ziris'
//* Create ZFS mount point for z/IRIS
//DEFINE EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DEFINE CLUSTER (NAME(hlq.ZIRIS.ZFS) -
LINEAR CYL(200 50))
VOLUMES(xxxxxx) -
/*
//* Mount the z/IRIS directory
//MOUNT EXEC PGM=IKJEFT01,DYNAMNBR=10
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
MOUNT FILESYSTEM('hlq.ZIRIS.ZFS') +
MOUNTPOINT('/usr/lpp/ziris') +
TYPE(ZFS) MODE(RDWR) /* AUTOMOVE */



Install and configure a z/IRIS client

  1. Extract and upload (in binary mode) the zirisv%%.pax to the $APP_HOME directory on z/OS USS. 

  2. Unpack the file using the tar command:

    pax command
    pax -w[v]f zirisv1%%.pax

    Use the -v option only if you want to see verbose output.

  3. Apply the authorized extended attribute to $APP_HOME/bin/CONSL001:

    extatttr command
    cd $APP_HOME/bin
    extattr +a ./CONSL001
  4. Make all commands in $APP_HOME/bin executable: 

    chmod command
    cd $APP_HOME/bin
    chmod +x ./*

z/IRIS License

The z/IRIS license file is provided separately and must be uploaded to the $APP_HOME directory.

ziris.comm.config

Configure the $APP_HOME/lib/ziris.comm.config to as required.

Parameters and options

Required parameters and options
# ziris.comm.config required parameters
# mode defines the technology used to stream SMF records.
#  syntax : mode={tcp!kafka}                              
mode=tcp

# host refers to the host address destination for SMF record streaming. 
#  If mode=tcp host is required and refers to the IronTap host address. 
#  If mode=kafka, host is a list of kafka bootstrap servers which overrides
#  all bootstrap servers listed in kafka producer.properties            
# syntax : host=[IronTap_host_address|kafka_bootstrap_servers]
host=irontap.mycompany.com:1337

# inmem is required and defines which active SMF in-memory resources z/IRIS will 
# read SMF records from. The inmem statement can be specified multiple times to
# configure multiple in-memory resources.
# syntax : inmem=smf.in.memory.resource.name                            
inmem=IFASMF.INMEM.SMF102                                          
# kafkaTopic defines the kafka cluster topic used to stream SMF records.
#  Required if mode=kakfa                                               
# syntax : kafkaTopic=kafka_topic_name                                  
kafkaTopic=smf   
Optional parameters
# ssl activates SSL for mode=tcp
# default : n (not active)      
syntax : ssl=y!n        
 
# kafkaConfig defines an alternate absolute path to a           
#  kafka producer.properties file. The default path is              
#  default : $APP_HOME/kafka/default-ziris-producer.properties  
#  syntax : kafkaConfig=/producer/properties/file               
kafkaConfig=kafka/default-ziris-producer.properties   
       
# timeout defines the timeout for SMF record streaming in seconds.
# timeout=ä0-2147483647ü                                          
timeout=7                                                              

# reconnect defines the number of seconds to wait before reconnecting the
#  stream after a timeout or error.                                     
# default : 7                                                           
# syntax : reconnect=ä0-2147483647ü         
reconnect=7

# monitor.thruput.interval defines an interval in seconds to display throughput
#  statistics. Only takes effect if com.msc.zos.ziris=FINE in logging.pr
# default : 60 seconds                                                  
# syntax : monitor.thruput.interval=ä5-3600ü                            
monitor.thruput.interval=5                                              

# alias is used to identify this client to the SMF receiver. 
# if mode=kafka, alias will overwrite the client.id Kafka Producer property                       
# alias=ziris_client_alias_name                   
alias=alias_name

 mode=kafka considerations...

Apache Kafka Producer properties when mode=kafka

The installation package contains default Apache Kafka Producer properties for z/IRIS streaming:

$APP_HOME/kafka/default-ziris-producer.properties.

Avoid making changes to the default-ziris-producer.properties as this may be overwritten by updates.

Instead create a new producer.properties file by :

  1. Copy the sample producer properties file in the $KAFKA_HOME/config directory.
  2. Edit the properties as required.
  3. Specify the new producer properties path and file name in the ziris.comm.config file using the kafkaConfig parameter.

For more information about required properties for z/IRIS SMF streaming, see Apache Kafka - Configuration Guide .


 Workload manager considerations...

Workload manager considerations

The z/IRIS started tasks should be allocated to low/discretionary priority in WLM.

Please consult your mainstorconcept technical professional if you require any assistance with amending your WLM policies.

Using z/IRIS clients

 Create a new z/IRIS proclib member ...

Create a z/IRIS client PROCLIB member

  1. Upload the ZIRISPRC started task procedure and the ZIRISJVM JZOS procedure files from zIRISzOSClient.v%.%.%.zip to a concatenated PROCLIB data set.

  2. Change $APP_HOME in JCL to the $APP_HOME directory created for z/IRIS in ZFS.
  3. If mode=kafka for SMF records streaming via a predefined and network accessible Kafka cluster, uncomment the following line and define the Kafka home directory to z/IRIS z/OS: 

    # export KAFKA_HOME=/usr/lpp/kafka/kafka_2.11-2.3.0
  4. Change $JVAHOME to the $JAVA_HOME directory where IBM Java version 8 resides.
  5. Save the PROCLIB member.

  6. Allocate the surrogate user id defined by the RACFJCL job: 

    RACF RDEFINE syntax example
    RDEFINE STARTED JOBNAME.** UACC(NONE) STDATA(USER(ziris) +
    GROUP(SYS1) TRUSTED(YES))
 Starting z/IRIS clients

Activating z/IRIS clients

To start the z/IRIS client Started Task, issue the following MVS Command:

Start z/IRIS started task syntax
/S ziris

where ziris is the member name of the z/IRIS PROCLIB member.


Please note:

When z/IRIS reports a failed connection during start-up, check that either the z/IRIS Linux server is active or that the Apache Kafka cluster is running with the accessible topics and brokers.

 Stopping z/IRIS clients...

Stopping  z/IRIS clients

To stop the started task issue the following MVS Command:

Stop z/IRIS started task syntax
/P ziris

Where ziris is the name of running z/IRIS STC Address Space.

 z/IRIS client modify commands...

z/IRIS client modify commands

z/OS users with SDSF and MVS Command Permissions, can interact with the z/IRIS client Started Task through MVS Modify Commands.
z/IRIS MVS Command Syntax :

Modify command syntax for z/IRIS started task
/F ziris,APPL=cmd

where:

ziris:  is the name of the z/IRIS STC
cmd:  is a valid z/IRIS Modify Command

HELP

lists and displays information about available commands:

z/IRIS HELP modify command
/F ziris,APPL=HELP

STATUS

displays a summary of current connections with SMF and the z/IRIS IronTap Linux Server.

z/IRIS STATUS modify command syntax
/F ziris,APPL=STATUS

Sample response:

Sample output from STATUS modify command
RESPONSE=CPAC      BPXM023I (SMFRAPI) ZIRIS will accept the STOP(P) and
RESPONSE=Modify(F) commands                                            
RESPONSE=CPAC      BPXM023I (SMFRAPI)                                  
RESPONSE=CPAC      ZIRIS STATUS Jul 18, 2018, 12:01:34 PM              
RESPONSE=CPAC      TCP Connection:  active to /172.25.999.999:9999     
RESPONSE=CPAC      Number of Listeners: 1                              
RESPONSE=CPAC      Listener 1 streaming from IFASMF.INMEM.SMFDB2  
 z/IRIS client logging...

z/IRIS client logging and error messages

  • STDOUT and STDERR are merged by default into SYSOUT. 
  • The Started Task will send messages to the SYSLOG when the messages require immediate attention, for example a TCP connection was lost and will try to reconnect as defined in the ziris.comm.config file.
  • Example SYSOUT from z/IRIS client Started Task:

    Example
    2020-03-26 09:34:24   INFO     z/IRIS Startup V1.3.0                       
    2020-03-26 09:34:28   INFO     ZIRISTST initializing System Properties       
    2020-03-26 09:34:28   INFO     ZIRISTST Checking License Information        
    2020-03-26 09:34:33   INFO     ZIRISTST Getting In-Memory Information...    
    2020-03-26 09:34:35   INFO     ZIRISTST Reading config File: /lib/ziris.comm.config
    2020-03-26 09:34:36   INFO     ZIRISTST Loaded Configuration...              

z/IRIS Features

Load Balancing & High Availability with Apache Kafka Integration

 Click here for more information...

Streaming SMF records through an intermediate predefined Kafka Cluster is now possible. Integration with Apache Kafka guarantees high availability and load balancing. 

Using an intermediate Kafka cluster provides the ability  to retain SMF records in the cluster for a configurable period and integrate the cluster into SMF record collection and eliminate traditional collection of SMF records using daily batch jobs on the mainframe. 

The configuration options and recommendations for z/IRIS processing can be found here:

Apache Kafka - Configuration Guide

z/OS work activity tracing 

 Click here for more information...

Common address space work activity SMF records (Type 30) provide the following information:

  • a batch job, APPC/MVS transaction program, OMVS forked or spawned address space starts
  • a batch job or step, APPC/MVS transaction program, OMVS forked or spawned address space ends
  • interval accounting for active jobs, started tasks, and system address spaces
  • TSO session starts or ends (user logon and logoff)

The following performance information is provided by SMF 30 records:

  • Work and batch job names and identifiers
  • CPU and specialty engine utilization(Service units and CPU time)
  • work start and end timestamps as well as duration/elapsed time
  • Completion codes which signal whether an error occurred during processing
  • Job Owners and assigned user ids

Benefits of streaming SMF 30 records in real-time:

  • z/OS work monitoring on modern APM platforms
  • Near real-time source for mainframe performance data which will increase the business value of existing mainframe capable infrastructure monitoring tools
  • Near real-time mainframe resource consumption charge back calculations possibilities
  • Incorporate mainframe administrators, developers and teams with modern DevOps and SysOps tools and processes

z/IRIS metric streaming 

 Click here for more information...

z/IRIS is capable of streaming metrics to DBMS instances that are used as data sources for metric monitoring software. 

Supported SMF records for z/IRIS metrics

  • IBM RMF SMF Type 70 records - CPU activity metrics

Support for additional SMF record types will be added incrementally and/or upon request.

Supported metric data sinks

  • InfluxDB 2.0

Support for alternate data sinks can be requested.

For more information about the metrics z/IRIS creates using IBM SMF records, see the z/IRIS metrics article.

Distributed Db2 for z/OS tracing 

 Click here for more information...

Db2 deadlock traces

Deadlock traces provide details regarding deadlock events that have occurred within a Db2 address space. z/IRIS processes and provides the application Holder and Waiter information from a deadlock event as trace-ids that are searchable within partner APM solution. This helps DevOps teams quickly locate applications holding Db2 resources as well as identify the applications who timed out due to the unavailability of these Db2 resources.

To enable this deadlock traces for z/IRIS processing, issue the following Db2 command:

Example
-START TRACE(PERFM) CLASS(32) IFCID(172) DEST(SMF) TDATA(CORRELATION) [SCOPE(GROUP)]

Db2 accounting traces 

Accounting information gives an overview of a unit-of-work completed for JDBC client application. Details include elapsed CPU seconds and response time, SQL requests summaries, amount of fetched data and commits or rollbacks.

To enable this accounting traces for z/IRIS processing, issue the following Db2 command:

Example
-START TRACE(ACCTG) CLASS(1) IFCID(3) DEST(SMF) TDATA(CORRELATION) [SCOPE(GROUP)]

Accounting trace details vary according to the classes applied when activating accounting tracing. Speak to your Db2 administrator to ensure your APM integration contains all the information your DevOps teams require for effective monitoring.

Please note:

To receive records for each Unit-Of-Work in the Db2 Address Space, the DB2 ZPARM ACCUMACC must be set to NO. If ACCUMACC is left active, the accounting records are rolled-up by the chosen aggregation field also specified in the ZPARM. Please see the IBM web pages for more information regarding the affected parameters:

  •  DDF/RRSAF ACCUM field (ACCUMACC subsystem parameter)
  •  AGGREGATION FIELDS field (ACCUMUID subsystem parameter)

Db2 performance traces

This Db2 performance traces provide low-level performance information. Activating these traces can affect Db2 for z/OS response time and CPU consumption, and should only be activated when required.

To enable this trace-set for to be processed by z/IRIS, the following Db2 Trace Command must be issued to the relevant Db2 Subsystem:

Example
-START TRACE(PERFM) CLASS(30) IFCID(58,59,60,61,62,63,64) DEST(SMF) TDATA(CORRELATION) [SCOPE(GROUP)]

Please note:

IBM does not recommend that users run this performance class traces in a production environment over extended periods of time.