Alert Troubleshooting Guide
Created , Updated
Problem: Transition from legacy alerts to enhanced alerting system
Grafana provides two different possibilities for alerts:
Legacy dashboard alerts which are deprecated and will be removed in Grafana 9
An enhanced alerting system which was introduced with Grafana 8
Legacy alerts are incompatible with Grafana’s enhanced alerting system. As consequence, if you want to use Grafana’s improved altering system, all existing legacy alerts need to be migrated.
Secondary issue: Grafana lacks support for template variables
Both alerting systems don’t support template variables which are used in some of our dashboards. This guide will recommend how to circumvent these limitations.
Legacy dashboard alerts
In general, we recommend to use Grafana’s enhanced alerting system. Some of our dashboards utilize template variables and transformations. Those need to be replaced in order to use these dashboards in conjunction with legacy dashboard alerts. This requires a considerable amount of work.
Transformations
In most cases, transformations are used in panels for naming purposes. We recommend to simply delete these transformations. A few panels use transformations for e.g. binary operations. Those panels need to be overhauled in such a way that they work without transformations.
Remove transformation from a panel
Template variables
Template variables can be configured in tab Variables of the Dashboard settings. They are used to dynamically query the configured data source to help visualizing its data. In order to use this dashboard in conjunction with legacy alerts template variables need to be removed from the dashboard entirely.
Location of template variables in a dashboard
Before doing that template variables have to be replaced in such a way that their initial purpose is maintained, at least in a static context. For that the type of a template variable needs to be considered.
Click on a template variable to find out its type
Query
We use a template variable of type Query to determine the possible values of a variable in a configured data source. For example, the template variable queueManager in our z/OS MQ Dashboard queries the data source to list all available queue managers. This enables the user to filter for a specific queue manager in the z/OS MQ Dashboard.
Example: Template variable queueManager and the resulting filter option in the z/OS MQ Dashboard
In order to remove a template variable of type Query follow the subsequent steps:
For every value, the chosen dashboard needs to be duplicated.
Export the duplicated dashboard as JSON file.
Replace all occurrences of ${<templateVariableName>} with a respective value in the JSON file.
Delete the former duplicated dashboard and import the modified JSON file in Grafana.
Remove the affected template variable of the imported dashboard.
Example: Replace ${queueManager} with one of its values (CSQ1)
Interval
As for now we only use one template variable of type Interval called ${windowPeriod} to define the aggregation window granularity for time series.
Example: Template variable windowPeriod and its configured options for aggregated window granularity
In order to remove the template variable ${windowPeriod} follow the subsequent steps:
Export the current dashboard as JSON file.
Replace all occurrences of ${windowPeriod} with v.windowPeriod or a time span of your choosing which is supported by Grafana(1s, 1m, 1h, 1d,…).
Delete the former dashboard and import the modified JSON file in Grafana.
Remove the template variable ${windowPeriod} of the imported dashboard.
Example: Replace ${windowPeriod} with v.windowPeriod
Constant
We use template variables of type Constant as placeholders for bigger mathematical terms, which allows us to adjust them for all concerned queries quickly and keep queries more human readable.
Example: Template variable offsetAlignment and its stored value
In order to remove a template variable of type Constant follow these steps:
Export the relevant dashboard as JSON file.
Replace all occurrences of ${<templateVariableName>} with its value in the JSON file.
Delete the former dashboard and import the modified JSON file in Grafana.
Remove the replaced template variable of the imported dashboard.
Example: Replace ${equidistantDuration} with its value
Enhanced alerting system
Fortunately, using the enhanced alerting system, alerts are no longer coupled with a specific dashboard. This means a dashboard that uses transformations or template variables doesn’t need to be modified. However, template variables can’t be used in alert queries and must be replaced with an appropriate value.
Example: Replace template variables queueManager and windowPeriod with suitable values