Created:  | Updated: 


z/IRIS OpenTelemetry Java Agent Instrumentation Extension BETA

OpenTelemetry users can activate the ziris-instrumentation-extension module with the experimental agent extension feature, which was introduced with the Java Agent release 1.3.0.
ziris-instrumentation-extension adds behaviour to the Java agent to facilitate context propagation for JDBC calls to Db2 z/OS servers, enabling Distributed Db2 for z/OS Observability with OpenTelemetry.

z/IRIS 1.6.3

Features

z/IRIS IronTap Container

Introducing z/IRIS IronTap as a Docker container. Simply run z/IRIS IronTap anywhere with a single command:

docker run \
--name irontap \
mainstorconcept.jfrog.io/ziris-docker-release/irontap:latest-kafka-instana
CODE

Find out more about running and configuring your dockerized z/IRIS IronTap.

Improvements

Db2 Accounting telemetry

  • The Db2 Accounting traces are now available with OpenTelemetry.

    • Db2 Accounting traces provide resource consumption and performance information within the z/OS system and Db2 subsystem like CPU (total, in-Db2), zIIP processor or durations (elapsed time, response time). See Db2 Accounting trace attributes with OpenTelemetry for detailed information.

Kafka record extractor tool

An easily configurable Kafka record extractor tool to extract SMF records from an Apache Kafka cluster for a specified time interval is now available. See Collecting SMF records from an Apache Kafka topic for more information.


z/IRIS v 1.6.2

Improvements

z/OS Connect telemetry 

  • An additional “system of record” (SOR) span is created for the services called by z/OS Connect allowing a more detailed end-to-end tracing and clearer visualization of latencies between the z/OS Connect REST API gateway and the SOR. See z/OS Connect Observability for more information.

  • Span status will be set to error for z/OS Connect timeouts and HTTP codes (400-502).

    • A useful error message will be provided.

Distributed Db2 deadlock telemetry

  • The Db2 deadlock traces are now available with OpenTelemetry.

    • Application traces are enriched with Db2 deadlock traces that contain information for root-cause analysis as well as identifiers to locate all applications impacted by the deadlock for simpler business impact analysis. See Db2 Deadlock trace attributes with OpenTelemetry for more information.

  • The values of the custom attribute zos.db2.lock.type (QW0172FR), which represents Db2 deadlock types, have been improved for better searchability.


z/IRIS v 1.6.1

Improvements

Integration of Apache Kafka

Apache Kafka 2.7.0 compatibility testing passed. Additionally, since version 0.10.2 Apache Kafka provides a bi-directional compatibility policy which guarantees that customers can upgrade their cluster brokers in isolation to updating the Kafka libraries used by the consumers in z/IRIS IronTap and producers in z/OS clients respectively.

IronTap

Logging - Customers can configure IronTap logging to ignore sensitive data, to prevent accidental security leaks when customers share logs outside their organization for support.


z/IRIS v 1.6.0

Features

OpenTelemetry integration

Observability and Monitoring requirements vary greatly depending on the business case and customer requirements. This is why a uniform standard is needed, that would allow different APM products to interact with each other, resulting in custom solutions that meet the needs of our customers. 

The Cloud Native Computing Foundation (CNCF) OpenTelemetry project is an observability framework that provides an open standard for telemetry data, including traces and metrics. The project has broad industry support and adoption from cloud providers, vendors and end users. With this release, the z/IRIS team acknowledges the importance of such an open standard, as its capabilities will ensure sustainable, vendor-neutral APM support for z/IRIS customers. 

Example of z/IRIS OpenTelemetry integration with Jaeger and z/OS Connect tracing

click to enlarge image

z/OS Connect telemetry 

z/IRIS enables observability and monitoring capabilities for client REST API requests that are processed by IBM z/OS Connect servers hosted on mainframe systems. Observability capabilities are available to partner and open-source APM vendors that support the OpenTelemetry Protocol (OTLP) specification.

The real-time processing of z/OS Connect SMF type 123 records provides two new features for mainframe-inclusive observability support for mainframe-backed DevOps:

  • Metrics streaming for SMF type 123 subtype 1 version 1 and version 2 

    z/OS Connect metrics are streamed to a preconfigured data sink that is integrated with visualization and analysis software and tools.  Official support for InfluxData's InfluxDB time series database is provided for all metric streaming features out-of-the-box. Additional adapters for streaming metrics to alternate data sinks or DBMS systems is available on request. See z/IRIS - z/OS Connect Metrics Streaming for more information.
    z/OS Connect metrics allow DevOps to monitor:

    • elapsed times (for all or selected z/OS Connect servers, by z/OS system or sysplex name),

    • latency between z/OS Connect servers and their System-Of-Records (SORs),

    • timeout frequency,

    • activity by user-name, service-name and types and more.

  • Trace streaming for SMF type 123 subtype 1 version 2
    DevOps teams have access to vital performance data and identifiers related to their application's REST-API requests that are processed by IBM z/OS Connect servers on mainframe systems. z/IRIS appends z/OS Connect spans to the trace-tree of the calling application in the APM user-interface. This is our first mainframe trace streaming feature that adheres to the (OTLP) specification. OTLP compliance ensures vendor-neutral support for our customers and future-oriented integration and support for compatible APM products.

Importable Grafana dashboard JSON

For quick start-up assistance, z/IRIS provides users with an importable JSON file, that creates customizable panels in a Grafana dashboard. Our zosconnect Grafana dashboard queries z/IRIS z/OS Connect Metrics that are stored in an InfluxDB bucket.

Example of the z/IRIS zosconnect Grafana dashboard

click to enlarge image


z/IRIS v 1.5.0

Features

CPU activity metrics (RMF SMF type 70 records)

Resource Measurement Facility (RMF) is IBM's strategic performance management product for z/OS and provides the ability to gather resource usage data with additional reporting and analysis functionality.
z/IRIS supports RMF SMF records, starting with record type 70, which provides CPU activity data. The RMF SMF records are streamed and parsed in real-time and integrated into open-source metric monitoring software.

z/IRIS RMF Metrics Streaming provides detailed documentation on the various metrics and data schemas created by z/IRIS.

InfluxDB integration support for metrics

z/IRIS supports streaming metrics to an InfluxDB 2.0 time series database for retention. InfluxDB instances can be used as data sources for analytics and visualization applications like Grafana. 

Streaming metrics to an alternate DBMS through z/IRIS can be requested.

Example of z/IRIS InfluxDB 2.0 RMF 70 schema

click to enlarge image

Grafana dashboard JSON

For quick start-up assistance, z/IRIS provides users with an importable .json file that creates the following panels in an RMF related Grafana dashboard:

Current in-ready work unit queue distribution per system

click to enlarge image

CPU contention per system

click to enlarge image

Mean CPU usage

click to enlarge image


z/IRIS v 1.4.2 ...

z/IRIS v 1.4.2

Improvements

Time zone customization

  • Configure z/OS time zones globally, per Sysplex or per LPAR. This improves the consistency of timestamps. The default time zone is UTC.

    • Old timestamp in local time:

      2020-10-06T12:34:49.930
      TEXT
    • New timestamp format with explicit UTC time zone:  

      2020-10-06T10:34:49.930Z
      TEXT

Instana APM integration

Db2 Performance

Db2 Performance spans provide detailed performance information sourced from Db2 for z/OS IFCIDs 58-66:

IFCID 

Instana call.name

IFCID 58

Db2 Performance : End Sql QW0058SN

IFCID 59

Db2 Performance: Fetch start QW0059SN

IFCID 60

Db2 Performance: QW0060ST start QW0060SN

IFCID 61

Db2 Performance: QW00612ST start QW0061SN

IFCID 62

Db2 Performance: QW0062ST start QW0062SN

IFCID 63

Db2 Performance: SQL statement

IFCID 64

Db2 Performance: QW0064ST start QW0064SN

IFCID 65

Db2 Performance: Open Cursor start QW0065SN

IFCID 66

Db2 Performance: Close Cursor start QW0066SN

For information on the IFCID fields used in Instana, please see "Custom tags in Instana APM".

  • Details regarding the Instana custom tags created from the performance class Db2 for z/OS IFCIDs can be found in "Custom tags in Instana APM".

  • Db2 Performance, Db2 Accounting and Db2 Deadlock spans are now Exit spans to prevent merging with the parent JDBC call spans while keeping all custom tags searchable via the Instana Analyze Dashboard.

  • Db2 Performance, Db2 Accounting and Db2 Deadlock spans are assign the Instana service.name Db2 and are of type DATABASE.

z/IRIS v 1.4.1 ...

z/IRIS v 1.4.1

Improvements

Distributed Db2 for z/OS traces for Instana APM

  • Instana traces created by z/IRIS for distributed Db2 for z/OS JDBC calls have been adapted to support Instana changes to custom trace processing.

    • "SQL related IFCIDs" call name has been changed and split into two new call names for improved search and filter capabilities.

      • Db2 Accounting (for IFCID 3 related performance information).

      • Db2 Deadlock (for IFCID 172 related Db2 resource and JDBC application data).

    • Call endpoints have been updated to point to the related Db2 for z/OS subsystem.

  • Enhance distributed Db2 for z/OS tags created by z/IRIS:

    • Db2 for z/OS related tags are prefixed with "zos.db2.*".

    • All tags have been updated for uniformity.

Deadlock Events for Instana APM
  • Each deadlock record (IFCID 172) processed by z/IRIS triggers a deadlock event in the Instana Events dashboard now.

  • Each deadlock event describes the impacted Db2 system, the number of Db2 resources affected and the Db2 for z/OS timestamp of the deadlock event. The trace.id, that contains all details regarding the deadlock, is provided in the event details.

Db2 Deadlocks

Db2 Deadlock calls provide the deadlock event, impacted resources as well as the holder and waiter sessions for each resource. Each child spans of the Db2 Deadlock trace provides the following information:

  • The Db2 for z/OS Resource Name (RNA) affected by the deadlock and the sessions holding-up or waiting for the RNA.

  • If the holder and waiter sessions (JDBC connections) are instrumented by Instana agents, z/IRIS will provide the relevant Instana trace.id under the following tags: zos.db2.deadlock.holder.trace.id and zos.db2.deadlock.waiter.trace.id. Use the Instana Analyze dashboard to locate the application calls identified by z/IRIS as participants of a deadlock.

  • Each Instana trace.id, identified by z/IRIS as a participant in the deadlock, is tagged as erroneous and enriched with the deadlock trace information required to assist DevOps with identifying all application calls affected by deadlocks in Db2 for z/OS.

  • z/OS and Db2 identifiers related to the holder and waiter JDBC calls, i.e: zos.db2.luw.id, zos.lu.name, zos.network.id, zos.system.name and zos.subsystem.name.

  • If an IPv4 address is available for holder and waiter sessions, z/IRIS will provide the network address under the following tags: zos.db2.holder.network.address and zos.db2.waiter.network.address.

Db2 Accounting

Db2 Accounting traces provide resource consumption and performance information within the z/OS system and Db2 subsystem, for JDBC calls, including:

  • In-Db2 CPU, zIIP processor and elapsed time (zos.db2.cpu.timezos.db2.ziip.time and zos.db2.elapsed.time).

  • Total CPU, zIIP processor and response time (zos.cpu.time, zos.ziip.time and zos.db2.response.time).

  • Service Units Factor required to calculate mainframe MSU costs for JDBC calls (zos.db2.su.factor).

  • Workload manager service class applied to the Db2 for z/OS JDBC call (zos.db2.workload.service.class.name).

  • z/OS and Db2 identifiers for the JDBC connection, i.e: zos.db2.luw.id, zos.lu.name, zos.network.id, zos.system.name and zos.subsystem.name.

Known issues

Instana APM integration

  • By default, Instana spans created by z/IRIS are delayed by 60 seconds to avoid unpredictable merging with parent spans. When merging occurs, certain span metrics are overwritten in the call tree, by z/IRIS span metrics  (i.e. endpoint and call name). Users can customize the delay as described in the 'z/IRIS - Configuring integration into Instana' article.

  • Start time for DB2 Accounting and Db2 Deadlock is derived from the SMF Header Timestamp and not from the DB2 Timestamps available in IFCID 3 or IFCID 172.

  • Instana "fixes" the timestamp provided by z/IRIS, which adjusts the start time of spans created z/IRIS. The issue has been reported to Instana and will be fixed in an upcoming release.

  • Instana custom tags are either string or boolean. Numeric-logical comparisons for tag values, are not yet available for custom traces in Instana.

z/IRIS v 1.4.0

z/IRIS v 1.4.0

Features

z/OS common address space work record streaming (SMF 30)

  • Parsing, normalizing and formatting of SMF Record Type 30, subtypes 1-6 in near real-time.

  • Intermittent storage in Apache Kafka cluster.

Instana APM z/OS common address space work traces

  • Traces are created when:

    • A batch job starts and ends.

    • A job step completes.

    • Interval accounting is active (frequency depends on the SMFPRMxx interval and synchronizing settings).

    • System address space accounting is active (frequency depends on the SMFPRMxx interval and synchronizing settings).

  • Error signal and event generation when steps and jobs end abnormally to alert DevOps teams timeously of any issues related to z/OS work.

  • All traces are enriched with custom tags to facilitate searching and filtering to easily locate mainframe traces for batch analysis.

Fixes

  • Corrected bugs related to path specifications in the ziris.comm.config file.

z/IRIS v 1.3.0 ...

z/IRIS 1.3.0

Features

Load Balancing & High Availability with Apache Kafka Integration

  • In order to guarantee High Availability and Load Balancing, streaming SMF records through an Apache Kafka Cluster is now possible.

  • z/IRIS IronTap subscribes to an Apache Kafka Cluster to consume SMF records.

  • The z/IRIS z/OS Client can be configured as an Apache Kafka Producer.

Improvements

z/IRIS IronTap Linux Server

  • z/IRIS IronTap can consume SMF records from Apache Kafka.

  • The following SMF data is now included in the span and tag section in Instana APM: 

SMF Field Name

Instana Tag

SMFxSID ( System identification from the LPAR SID parameter)

SystemId

SMFxSSI (Subsystem identification value set by the SUBSYS=option specified in the SMF macros)

SubsystemId

  • z/IRIS IronTap metrics are visible in Instana:

    Click to enlarge IronTap metrics in Instana APM

z/OS Client

  • All z/OS Client logging is now controlled over the java.util.logging api. The default properties are provided in the $APP_HOME/lib/logging.properties file.

  • Throughput logging can be activated using the monitor.thruput.interval ziris.comm.config parameter. SMF Record throughput statistics will only be written to STDOUT when the logging level set to FINE as follows in logging.properties:

    ziris.comm.config extract example

    # monitor.thruput.interval defines an interval in seconds to display throughput
    #  statistics. Only takes effect if com.msc.zos.ziris=FINE in logging.properties
    # default : 60 seconds                                                  
    # syntax : monitor.thruput.interval={5-3600}                            
    monitor.thruput.interval=5                                              
    CODE

    logging.properties logging level for throughput monitoring

     com.msc.zos.ziris=FINE
    CODE

Fixes

  • Stabilized z/OS Client reconnect feature for mode=tcp.

Reported issues

  • None

z/IRIS 1.2.1 and earlier releases...

z/IRIS v 1.2.1

Improvements

IronTap Linux Server

The names of the following fields have been changed to improve readability. The values were converted into CPU Seconds: 

Old name

New name

CPU End Time

(not displayed, used to calculate "Total CPU Seconds")

CPU Start Time

(not displayed, used to calculate "Total CPU Seconds)

CPU Time Executing DB2

Total MVS

DB2 Processing of Accel. Eligible SQL

Accelerator SQL on CP

DB2 Processing on ZIIP of Accel. Eligible SQL

Accelerator SQL on ZIIP

In-DB2 on ZIIP

In DB2

Main Appl. Triggers

Appl. Triggers

Nested Task Triggers on ZIIP

Nested Triggers

On ZIIP Total

Total

SP on Std CPU Class 1 and 2

Appl. SP

SP on ZIIP Class 1 and 2

Appl. SP

SP Requests on ZIIP in-DB2

SP in WLM

SP Sql Stmts in WLM Address Space

SP SQL

SP SQL Stmts on ZIIP in-DB2

UDF in WLM

Trigger Exeution on ZIIP

Appl. Trigger

UDF in SP on ZIIP

UDF in WLM 2

UDF in-DB2 Class 1 and 2

Appl. UDF

UDF in-DB2 on ZIIP

In Db2 UDF in WLM

UDF on ZIIP CPU

Appl. UDF

UDF processed in SP

UDF

WLM SP Address Spaces

SP

ZIIP Eligible on Std CPU

On CP

n/a

Total CPU Seconds

n/a

UDF SQL


z/OS Client

  • Improved decision branching for streaming modes (in preparation for ETL streaming option between client and server).

    • Message flooding prevention when reconnecting after a TCP connection/IO Failure. The z/OS Client will log 5 reconnect attempts and thereafter logging for the reconnect will be suppressed until a connection has been established. The following message will be logged in the STDOUT DD:

Example
Jul 15, 2019, 10:31:41 AM ! ZIRISUAT 1 error reconnecting via TCP, EDC8128I Connection refused. (Connection refused)
Jul 15, 2019, 10:31:41 AM ! ZIRISUAT 1 Reconnect failed, automatic retry every 10 seconds.
                         Reconnect logging will be SUPPRESSED until a successful connection has been established
CODE

Fixes

  • None

Reported issues

  • None


z/IRIS v 1.2.0

Major Features include:

  • Db2 LUWID correlation and ordering.

  • Detecting Stored Procedure metrics and tagging for Instana APM users.

  • Detecting Deadlock metrics and tagging for Instana APM users.

Improvements:

  • Instana APM - Response Time alignment to Instana standard (<1ms).

  • Simplified SQL Communication Area Output.

Fixes:

  • IFCID 3 parsing corrections between v11 and v12.

  • Statement Type in IFCID 63 - offset corrected to parse the self defining section.

  • Timestamp corrections.

Reported Issues

  • No reported issues


Accessing z/IRIS Software

All software packages and container images can be retrieved from the mainstorconcept JFrog repository. 

Please note:

Licenses are obtained seperately. Contact technical support or your z/IRIS representative to for more information.