Created: | Updated:
Enable mutual TLS (mTLS) for OTLP Endpoints
z/IRIS is now able to send one or more trusted client certificates to OTLP endpoints for authentication in environments where bidirectional authentication is required. Refer to Configuring IronTap OpenTelemetry integration for instructions on enabling mTLS for IronTap’s communication with OpenTelemetry.
Distributed MQ to CICS for z/OS Workflow Tracing BETA
Tracking message processing between distributed MQ clients and CICS transactions is now available.
When z/IRIS identifies that a CICS transaction processed MQ messages from upstream clients, the context of the upstream client is added to the transaction's CICS Spans. This instructs the observability back-end to correlate the z/IRIS CICS Span to the client application's trace.
Users are able to ascertain which distributed applications utilize MQ for z/OS queues and queue managers for communication with CICS transactions. Furthermore, users can analyze the latency, call rate and error rate of CICS and MQ alongside the rest of the technology stack for digital business services.
Below is an example of a trace composed of OTEL spans from various HTTP, and JMS applications on Linux, supplemented by CICS and MQ for z/OS spans created by z/IRIS for applications running on mainframe:
The Software Bill of Materials (SBOM) is a critical component in ensuring software security and effective supply chain risk management. Because mainstorconcept values the security of its applications post deployment, a comprehensive SBOM for z/IRIS IronTap server packages and docker images is now available for auditing via tools like vulnerability scanners.
Enable TLS for OTLP Endpoints
With this release, users now have the capability to employ trusted certificates to verify the TLS credentials of an OTLP endpoint. This empowers users to establish a secure connection between z/IRIS and an OTLP endpoint, such as the OpenTelemetry Collector.
Db2 Accounting spans renamed to DRDA | SQL
Tracing Db2 for z/OS processing was the pioneer trace telemetry delivered by z/IRIS. Initially, mainframe users were content with these spans being referred to as
Db2 Accounting, as this referenced the IFCID 3 data from which the spans are sourced. However, z/IRIS places great value on providing mainframe telemetry in a format that can be understood by mainframe and non-mainframe users alike. Therefore, the Db2 Accounting spans have been renamed to indicate the unit-of-work being processed.
The new span name format is:
db2-client-connection-type + db2-location-name
db2-client-connection-type is one of the following:
Distributed Relational Database Architecture (DRDA) facilitates the distribution of relational data across multiple platforms and is employed for client connections through APIs, JDBC, and TCPIP. To enhance clarity, Db2 spans associated with units of work initiated by these client connections will be prefixed with
Structured Query Language is used by local applications to access Db2 resources. To ensure easy identification, Db2 spans representing SQL calls processed on Db2 for z/OS will now be prefixed with
SQLfor improved clarity.
db2-location-name references the Db2 for z/OS location that processed the unit of work.
Adaptation to the current semantic conventions of OpenTelemetry
With this release, span attributes have undergone updates to align with the new semantic conventions of OpenTelemetry, while also introducing additional attributes. For this, consider the Spans page and all its subpages.
The attribute changes are listed below:
New attribute key:
New attribute key:
New attribute key:
net.host.namewith default value
localhostor the specified host name of the z/OS Connect server.
New attribute key:
net.host.portwith the port extracted from
z/IRIS z/OS Client IBM Semeru 11 support
This release provides a new z/OS Client that now supports the IBM Semeru Runtime Certified Edition for z/OS 11.
This release extends z/IRIS Mainframe Workflow Tracing, which provides transaction tracking within z/OS, so that users gain more insights into back-end processing on mainframe.
z/OS Connect to CICS Tracing
Before this update, users could only identify z/OS Connect APIs backed by CICS by analyzing data available in the System-of-Record (SOR) child span for z/OS Connect traces. However, with the latest release of z/IRIS, a new CICS Transaction child span is included in the REST API request trace. This child span provides valuable transaction, program, performance, and resource-related data for the CICS call, resulting in faster root-cause identification and improved analysis through supplemented CICS identification and performance metrics.
See z/OS Connect Observability | z/OS-Connect-to-CICS for more information about z/OS Connect and CICS Transaction workflows.
CICS to Db2 Tracing
z/IRIS now adds a Db2 child span to a CICS Transaction trace whenever CICS transactions execute Db2 calls. This new correlation context exposes the dependencies between CICS transactions and Db2 resources on z/OS, enabling DevOps engineers to obtain a deeper understanding of the performance of mainframe applications in digital business services.
For further information about CICS and Db2 SMF record correlation, see CICS Transaction Observability | CICS to Db2.
All features created in this release culminate in users obtaining end-to-end workflow and performance insights where requests depend on IBM MQ for z/OS and CICS transactions. Find out more about these features below.
MQ for z/OS Trace
Traces are created from SMF 116 type 1 & 2 records to depict tasks/threads that initiated within a z/OS MQ Queue Manager to process messages in queues or topics. Each MQ for z/OS trace identifies how the task was initiated, the latency of the task, and whether the Queue Manager detected any errors. Child spans within the trace each identify an object (queue or a topic) that the task used to process messages. The time spent connected to the object is shown through the child spans' duration.
CICS Transaction spans
Spans are created from SMF 110 records to describe CICS transaction executed on z/OS. Each span identifies the transaction, its response time. Also contained within the spans are Resource and Trace attributes. Resource attributes identify the CICS subsystem as well as the z/OS LPAR and Sysplex that processed the transaction. Trace attributes provide additional performance and resource utilization metrics for the processing of the transaction.
Distributed MQ to MQ for z/OS Workflow Tracing BETA
A new function enables the correlation of APM traces from distributed IBM MQ clients with MQ for z/OS spans created by z/IRIS. This correlation reveals which client applications depend on MQ for z/OS queues and queue managers for digital business services. The insight provided by MQ for z/OS spans will help users identify how mainframe resources are consumed and how mainframe applications perform for distributed applications.
Below is an example of a trace composed of spans from various HTTP, and JMS applications on Linux, supplemented by MQ for z/OS spans from z/IRIS:
Mainframe Workflow Tracing for IBM MQ and CICS spans BETA
Users can identify CICS Transaction that use MQ for z/OS in a single logical unit of work. This enables visibility into the relationship between these services and enables holistic analysis and monitoring for users.
Below is an example of a trace composed of CICS and MQ for z/OS spans for a single transaction:
z/IRIS Mainframe Workflow Tracing for z/OS Work Batch Jobs BETA
This release supports correlating multiple SMF Type 30 records. As a result, z/IRIS users will get a single flame graph, composed of spans that each represent the phases and steps of a batch jobs.
See example below:
Clock Skew correction for mainframe spans TEST | POC
Differing system clocks diverge when not adequately synchronized. This “clock skew“ results in misaligned time-based data that deteriorates user experience on observability platforms and complicates analysis.
If clock synchronization over Network Time Protocol(NTP), System Time Protocol (STP)and/or an external time source is unavailable while trialing z/IRIS, it will impact the way relationships between distributed and mainframe applications are depicted. For testing purposes only, a clock skew correction can be configured in IronTap to simulate synchronized time between mainframe and open-systems. See Configure IronTap server for more information about the
z/IRIS Mainframe Workflow Tracing for z/OS Connect and Db2 records BETA
This release contains the first version of the z/IRIS Mainframe Workflow Tracing, allowing z/IRIS to correlate multiple SMF records into meaningful spans and traces. The scope of this release encompasses our first use case, which is the correlation of related z/OS Connect and Db2 SMF records. This allows users to see true end-to-end workflows of traces involving these two mainframe services. See example flame graphs for Datadog and Jaeger below:
See z/IRIS Mainframe Workflow Tracing to find out more about the z/IRIS SMF Correlation feature.
z/OS Connect Workflow Tracing contains details about the z/OS Connect and Db2 SMF correlations.
An issue causing warning logs regarding missing configuration parameters has been resolved.
🌟 Datadog API integration support BETA
Datadog APM Traces
Datadog users can elect to stream traces via the Datadog Agent API. This integration is provided in addition to the existing OpenTelemetry support, to accelerate proof of concept/value projects for Datadog users.
Please refer to Configuring Irontap Datadog integration to find out more about using Datadog APIs to enable mainframe-inclusive observability.
Datadog API integration is currently BETA. Technical support is provided to selected customers only. Contact mainstorconcept support should your organization be interested in testing Datadog Trace API integration.
🎉 Datadog Events
To ensure DevOps teams can proactively address anomalies and errors to avoid major incidents and outages requires adequate visibility into application issues that may impact overall business performance. z/IRIS creates events that leverage Datadog’s Unified Tagging capabilities. Datadog’s Events Explorer is a powerful user interface to search, analyse, and filter events from any source in one place.
Please refer to Configuring Irontap Datadog integration for guidance on how to enable Irontap to create Datadog Events using the Events API.
🎉 z/OS MQ Metrics 🎉
IBM MQ plays a critical role in business application communication, this is why maintaining performance and resource availability for MQ components like queue managers and buffer pools is crucial.
Anomaly detection, issue analysis and health insights for your MQ systems facilitates planning, optimization and reliability efforts.
In this release, z/IRIS introduces z/OS MQ Metrics, that are streamed in real-time to ensure instantaneous access to performance indicators that signal changes in your MQ infrastructure, applications, or client behavior.
MQ statistics records (SMF Type 115) contains a multitude of statistics from various resources within the system. This release of z/OS MQ Metrics focuses on the most vital performance indicators for monitoring, analysis and alerting purposes. Customers may request support for additional fields contained in SMF 115 records by contacting support.
Grafana z/OS MQ Dashboard
Importable Grafana z/OS MQ Dashboards are designed to provide quick start assistance for users, by visualizing and summarizing the key performance indicators provided by z/OS MQ Metrics. To meet your organization’s requirements for enterprise monitoring, users can customize panels and thresholds, as well as create alerts.
Datadog z/OS MQ Dashboard and Monitors
Datadog users are provided with importable dashboards and monitors that leverage Datadog's powerful APM capabilities, enriching your day-to-day enterprise monitoring activities, and removing mainframe monitoring silos.
Common span attributes
os.type are now part of every z/IRIS span to improve overall usability.
Db2 Accounting Traces
Trace semantics has been adapted. The span’s duration is now set to the accounting elapsed time calculated via
zos.db2.end.timestamp - zos.db2.start.timestamp (
z/OS Connect Metrics
The z/OS Connect metric's naming convention has been unified for SMF 123 subversion 1 and 2 to simplify their usage, enabling the user to filter more easily and create customized dashboards faster. In addition, the list of measurement for z/OS Connect has been expanded to include the zos.connect.timed_out.requests and zos.connect.successful.requests. See z/OS Connect Metrics Streaming for more information.
RMF metrics have been revamped, enabling more flexible querying as well as ensuring compatibility with telemetry pipeline processors (e.g. OpenTelemetry’s Contrib Collector). See RMF Metrics Streaming for more information about how RMF metrics have been overhauled, and take a look at how an OpenTelemetry Contrib Cmore informationollector can be configured.
New z/OS Infrastructure Dashboard
RMF gathers data about z/OS resource usage and provides reports of system and sysplex. Using this dashboard, you can get a high-level view and understanding of your performance, and resource utilization on the mainframe. The RMF metrics are used to populate our z/OS Infrastructure Dashboard. In order to keep an eye on your performance and activity in mainframe you can track your CPU contention and mean usage by LPAR and MVS. Additionally, you obtain an overview of your In-Ready-Work-Unit-Queue distribution.
z/OS Infrastructure Dashboard in Datadog
Dashboards optimized for Grafana 8
z/OS Connect and z/OS infrastructure dashboards are now available for Grafana 8. DevOps teams can monitor and analyze how z/OS Connect and mainframe service providers perform for REST API requests. z/IRIS users can identify latencies and errors that impact REST API requests hosted and processed on mainframe systems. In addition, the z/OS infrastructure Dashboard provides information on mean usage, performance and activity, as well as In-Ready-Work-Unit-Queue distribution.