Add Severities to template data and hasString to template function

[jkroepke]

Motivation of this PR

Some receiver of Alertmanager handling a persistent state of alerts.

Looking at OpsGenie, it can be setup one alert with group of alerts.

However, currently its not really possible to get the highest severity of the alert group. One potential solution is having severity as part of the group_labels, but this results into distinct alerts, because the HashKey of an alert group changes and grouped alerts may get splited.

In conclusion, I added a new template data field severities which hold a list of string contains all values of the label severity. In addition, I added a new template func hasString that maps to slices.Contains.

The result would be a template that could be used to define a OpsGenie priority:

{{ if hasString "critical" .Severities }}P1{{ else if hasString "warning" .Severities }}P3{{ else }}P5{{ end }}

It's also useful for #3590, since the receiver has the capability to update existing issues and it would be great if the priority can be updated as well.

[grobinson-grafana]

Hi! 👋 Alertmanager doesn't have severity, and Prometheus users are not required to set use a severity label, so I'm not sure if this is something that will be accepted. Is it not possible to template the severity for Opsgenie using the existing templating infrastructure?

[jkroepke]

Hey!

Alertmanager doesn't have severity, and Prometheus users are not required to set use a severity label,

I know. In that cases, the list is just empty. It won't break anything. While AM doesn't have severity, it would be great if AM can assists users they are using commonly-known pattern.

Is it not possible to template the severity for Opsgenie using the existing templating infrastructure?

If you show me, how this is possible, we can close the PR.

What I need is loop through all alerts, look for a specific. I tires something like

{{- range .Alerts.Firing -}}
{{- range .Labels.SortedPairs -}}
{{- if and (eq .Name "severity") (eq .Value "critical") -}}P1{{- end -}}
{{- end -}}
{{- end -}}

would result into P1P1P1P1.

In Helm templating engine, something like this is possible:

{{ $priority := "P3" }}
{{- range .Alerts.Firing -}}
{{- range .Labels.SortedPairs -}}
{{- if and (eq .Name "severity") (eq .Value "critical") -}}
{{- $_ := set $priority "P1" . -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{ $priority }}

But the set function is not available in AM.

Pulling https://github.com/Masterminds/sprig may resolve this issues, it allows to do crazy things inside go template. Looking at #3770 it unclear, if this is getting accepted. The current approach show me that creating functions that are required would be the way to go.

[jkroepke]

If this is too severity specifc, what about an alternative solution that provides all possible label values?

Like an variables, map[string][]string, which has label names as keys and all possible label values as label

Then, in go template this would be possible:

{{ if hasString "critical" .AlertLabels.severity }}P1{{ else if hasString "warning" .AlertLabels.severity }}P3{{ else }}P5{{ end }}

but alternatives are possible, too:

{{ if hasString "PROD" .AlertLabels.env }}P1{{ else }}P5{{ end }}

[grobinson-grafana]

What about this?

{{- range .Alerts.Firing -}}
{{- if (eq (index .Labels "severity") "critical") -}}P1{{ break }}{{- end -}}
{{- end -}}

[jkroepke]

{{ break }} - nice, I was not aware of that.

But the issue with that, it give P1 back if there is an alert with severity=critical. If not, then nothing is returned.

{{- range .Alerts.Firing -}}
{{- if (eq (index .Labels "severity") "critical") -}}P1{{ break }}{{- end -}}
{{- end -}}
{{- range .Alerts.Firing -}}
{{- if (eq (index .Labels "severity") "warning") -}}P2{{ break }}{{- end -}}
{{- end -}}

result into P1P2, if there are multiple alerts with different severities.

In native go, I would use return statement instead break, but it does not seems available.

[grobinson-grafana]

What about this?

{{- define "priority" -}}
{{- $priority := "" }}
{{- range . -}}
{{- $severity := index .Labels "severity" -}}
{{- if (eq $severity "critical") -}}
{{- $priority = "p1" -}}
{{- else if (and (eq $severity "warning") (eq $priority "")) -}}
{{- $priority = "p2" -}}
{{- end -}}
{{- end -}}
{{- $priority -}}
{{- end -}}

You can then execute the template with:

{{ template "priority" .Alerts.Firing }}

[jkroepke]

I'm impressed, thanks!

{{- define "priority" -}}
    {{- $priority := "" }}
    {{- range .Alerts.Firing -}}
        {{- $severity := index .Labels "severity" -}}
        {{- if (eq $severity "critical") -}}
            {{- $priority = "High" -}}
        {{- else if (and (eq $severity "warning") (ne $priority "High")) -}}
            {{- $priority = "Medium" -}}
        {{- else if (and (eq $severity "info") (eq $priority "")) -}}
            {{- $priority = "Low" -}}
        {{- end -}}
    {{- end -}}
    {{- if eq $priority "" -}}
        {{- range .Alerts.Resolved -}}
            {{- $severity := index .Labels "severity" -}}
            {{- if (eq $severity "critical") -}}
                {{- $priority = "High" -}}
            {{- else if (and (eq $severity "warning") (ne $priority "High")) -}}
                {{- $priority = "Medium" -}}
            {{- else if (and (eq $severity "info") (eq $priority "")) -}}
                {{- $priority = "Low" -}}
            {{- end -}}
        {{- end -}}
    {{- end -}}
    {{- $priority -}}
{{- end -}}