[Feature]: Honor severity label for firing alerts and include annotations in output

[Inderdeep01]

Describe the feature request

Description

Currently, check_prometheus determines the Icinga exit status based solely on the alert state:

• firing → CRITICAL (exit 2)
• pending → WARNING (exit 1)
• inactive → OK (exit 0)

This ignores the severity label that Prometheus/Alertmanager/vmalert rules commonly use to indicate the intended severity level. As a result, firing alerts with severity=warning are reported as CRITICAL in monitoring systems like Icinga, causing unnecessary noise and incorrect escalations.

Additionally, the output only includes labels but not annotations (summary, description), which are often essential for on-call engineers to understand and triage alerts quickly.

Problem

Example current output:

[CRITICAL] - 17 Alerts: 17 Firing - 0 Pending - 0 Inactive
_ [CRITICAL] [XXXXXServiceLowHealthyRatio] is firing - value: 0.00 - {...,"severity":"warning"}

The alert has severity=warning in its labels, but the plugin returns CRITICAL because the alert is firing.

Impact:

• Warning-level alerts create CRITICAL noise in monitoring systems
• Paging and escalation paths don't align with actual alert severity
• Alert annotations (summary/description) are not visible in the output

Proposed Solution

1. Map severity label to exit status for firing alerts

For firing alerts, check the severity label and map it to the appropriate exit code:

Severity Label	Exit Status
critical (default)	CRITICAL (2)
warning, warn	WARNING (1)
info, informational	OK (0)

The lookup should check alert-level labels first, then fall back to rule-level labels.

Proposed change to internal/alert/alert.go:

func (a *Rule) GetStatus() (status int) {
    state := a.AlertingRule.State

    switch state {
    case string(v1.AlertStateFiring):
        status = check.Critical
    case string(v1.AlertStatePending):
        status = check.Warning
    case string(v1.AlertStateInactive):
        status = check.OK
    default:
        status = check.Unknown
    }

    // Honor severity label for firing alerts
    if state == string(v1.AlertStateFiring) {
        severity := ""
        if a.Alert != nil {
            if v, ok := a.Alert.Labels["severity"]; ok {
                severity = strings.ToLower(string(v))
            }
        }
        if severity == "" {
            if v, ok := a.AlertingRule.Labels["severity"]; ok {
                severity = strings.ToLower(string(v))
            }
        }
        switch severity {
        case "warning", "warn":
            return check.Warning
        case "info", "informational":
            return check.OK
        case "critical":
            return check.Critical
        }
    }

    return status
}

2. Include annotations in output

Append summary and description annotations to the alert output for better context:

// In GetOutput(), after writing labels:

// Append common annotations for clarity in downstream UIs.
if summary, ok := a.Alert.Annotations["summary"]; ok {
    out.WriteString(fmt.Sprintf(" - summary: %s", strings.ReplaceAll(string(summary), "\n", " ")))
}
if description, ok := a.Alert.Annotations["description"]; ok {
    out.WriteString(fmt.Sprintf(" - description: %s", strings.ReplaceAll(string(description), "\n", " ")))
}

Expected Output After Change

[WARNING] - 17 Alerts: 17 Firing - 0 Pending - 0 Inactive
_ [WARNING] [XXXXXServiceLowHealthyRatio] is firing - value: 0.00 - {...,"severity":"warning"} - summary: Healthy ratio below threshold - description: Service healthy ratio is 0%

Alternatives Considered

1. CLI flag to enable severity mapping - Could add --honor-severity flag to make this opt-in for backwards compatibility

Use Case

We use vmalert to evaluate Prometheus-style alerting rules and surface them in Icinga using check_prometheus. Our alerting rules use severity=warning for non-critical alerts that should not page on-call engineers. Without this change, all firing alerts appear as CRITICAL in Icinga regardless of their intended severity.

Additional Context

This follows the common pattern used by Alertmanager and most Prometheus alerting setups where the severity label indicates the intended alert level. Supporting this label would make check_prometheus more compatible with standard Prometheus/Alertmanager workflows.

I have a working implementation of this and can submit a PR if you're interested.

[martialblog]

Hi, thanks for the detailed issue! In general I like the idea.

The only issue I see with it is that neither the "severity" label nor its content can be expected.
Say we cover the described cases ("warning", "warn", "info", "informational", "critical") then users with other naming schemes have issues (e.g. trace, debug, fatal, and so on).

Maybe we can make this proposal more general with some CLI flags? I'm note sure myself what a good solution could be, I'll think about it.

We used the alert state since it's something that is expected to exist and the values are fixed.

Regards,
Markus

[Inderdeep01]

Hey Markus, thank you for your thoughts and consideration.

You make a valid point about the label name and values not being standardized.

In the meantime, I wanted to share that the implementation I described has been working well for our use case. We run vmalert + Icinga with the common severity=warning/critical convention, and the patch significantly reduced alert noise. Happy to contribute in future if there's a consensus on the approach.

Thanks again!
Regards,
Inderdeep Singh

[martialblog]

Maybe we could have a flag to make this more flexible something like --label-exit-state serverity=warn=1 -> labelKey = LabelValue = Resulting ExitCode when this alert is firing.

Might be a bit convoluted and needs proper error handling. Just my first idea.

[Inderdeep01]

i think we can merge those two ideas. we can choose the default keywords for the defined(widely used like warning/warn or some other you have seen) severities, but with the option to override those keywords. so it will be either use --honor-severity with no flags or --label-exit-state with the flags.
that way it might be possible to have a standard with the option to customize for more strict use cases. open to your views.

[martialblog]

Did a little prototype with the new flag: https://github.com/NETWAYS/check_prometheus/tree/feature/label-status

Still a bit hacky.

[RincewindsHat]

Maybe we could have a flag to make this more flexible something like --label-exit-state serverity=warn=1 -> labelKey = LabelValue = Resulting ExitCode when this alert is firing.

Might be a bit convoluted and needs proper error handling. Just my first idea.

Agree with the convoluted argument, how about --warning-label severity=warn --warning-label severity=warning --critical-label severity=crit --critical-label severity=catastrophic --label-unknown severity=ok and so on? The last one would be the catch-all for missing mappings.
I don't know much about common layouts, but if there is something like a standard with a sizable adaption, it would be nice if that would be built in.

[martialblog]

As far as I know there's no "standard". Many examples use "severity", but there is no convention for the values. Some examples use "warning/critical/info" others use something like "page".

I mean, we could also have a convention here in the plugin that Prometheus users can use to interact with the plugin:

  - alert: HighRequestLatency
    expr: job:request_latency_seconds:mean5m{job="myjob"} > 0.5
    for: 10m
    labels:
      icingaState: warning

All you would have to do is add another label to your existing alertrule and the plugin handles it.

[martialblog]

Did a little prototype here https://github.com/NETWAYS/check_prometheus/tree/feature/state-label

I kinda like this solution. It's simple to implement, it does not interfere with existing labels on AlertRules (since people might already have different values for the severity label and dependencies on the values downstream somewhere).
It costs nothing to add another label to the AlertRule.

EDIT: We could even have a flag to override the label-key.

[martialblog]

Done with #118