group_by: [alertname, alertstate] doesn't work as it should

[Drugoy]

What did you do?
I used

group_by:
  - alertname
  - alertstate

config and got multiple alerts (both firing and resolved) mixed together in a single notification.

What did you expect to see?
Separate notifications: one for firing alerts and the other for resolved ones.

What did you see instead? Under which circumstances?
A single notification with multiple alerts (both firing and resolved) mixed together.

Environment
Yes.

• System information:

Linux 3.10.0-1127.el7.x86_64 x86_64

• Alertmanager version:

alertmanager, version 0.21.0 (branch: HEAD, revision: 4c6c03e)
build user: root@dee35927357f
build date: 20200617-08:54:02
go version: go1.14.4

• Prometheus version:

prometheus, version 2.19.2 (branch: HEAD, revision: c448ada63d83002e9c1d2c9f84e09f55a61f0ff7)
build user: root@dd72efe1549d
build date: 20200626-09:02:20
go version: go1.14.4

• Alertmanager configuration file:

global:
  resolve_timeout: 5m
  smtp_from: x@y.com
  smtp_smarthost: 10.0.0.2:25
  smtp_require_tls: false

route:
  group_by:
    - alertname
    - alertstate
  group_wait: 30s
  group_interval: 1m
  repeat_interval: 3h
  receiver: topguys

receivers:
  - name: topguys
    email_configs:
      - to: z@y.com
        send_resolved: true

[chaishiqi]

I have came across the same issue,wait for an answer T_T

[roidelapluie]

Alertstate is not a label sent to alertmanager. Alertmanager will always send resolved and active alerts in the same notification.

[chaishiqi]

Alertstate is not a label sent to alertmanager. Alertmanager will always send resolved and active alerts in the same notification.

Got it. Thanks for your answer.Will it be sperated by reslove and alert in the future?

[roidelapluie]

The goal of alertmanager is to send as few notifications as possible; so that seems unlikely.

You can use group_by: [...] to not have any grouping at all if you wish so.

[Drugoy]

That goal needs to be re-evaluated then.
I'd see the main goal for alertmanager to send alerts that are maximally useful to the recipients and comfortable to work with.
Using no grouping at all is even less comfortable.

Is it due to difficulty of implementing alertstate label or is it more of a political decision not to implement it?

[roidelapluie]

What the alertmanager is doing is sending firing alerts first, then resolved alerts, all grouped.

If we split by alert state, if I receive a email with resolved alerts, I do not know if it is all resolved or only part is resolved and the other notification is lost (because the receiver could not handle both at the same time, or for some reason the notification was rejected).

It is also unclear how that should behave with the different timers we have now.

[Drugoy]

or only part is resolved and the other notification is lost (because the receiver could not handle both at the same time, or for some reason the notification was rejected).

It's not alertmanager's job to think of whether the notification was lost along the way and this problem has nothing to do with splitting alerts by state: it would cause you as much harm (if not more) if your notification with grouped alerts got lost.

if I receive a email with resolved alerts, I do not know if it is all resolved or only part is resolved

That problem is actual now as well: I got a bunch of notifications. Each of them has grouped alerts.
How would I know if there are any firing alerts left?
(I think I should file a separate ticket with that feature request)
I think a good solution would be introduction of a counter of currently still firing alerts so that 'resolved' notifications could look like 5/7 alerts resolved.

[roidelapluie]

I got a bunch of notifications. Each of them has grouped alerts.
How would I know if there any firing alerts left?

If the first notification is solved, there are no firing alert left.

You can create the counter in templates.

[Drugoy]

Alertstate is not a label sent to alertmanager. Alertmanager will always send resolved and active alerts in the same notification.

Hold up, but https://prometheus.io/docs/alerting/latest/configuration/#webhook_config says the alert structure is:

{
  "version": "4",
  "groupKey": <string>,              // key identifying the group of alerts (e.g. to deduplicate)
  "truncatedAlerts": <int>,          // how many alerts have been truncated due to "max_alerts"
  "status": "<resolved|firing>",
  "receiver": <string>,
  "groupLabels": <object>,
  "commonLabels": <object>,
  "commonAnnotations": <object>,
  "externalURL": <string>,           // backlink to the Alertmanager.
  "alerts": [
    {
      "status": "<resolved|firing>",
      "labels": <object>,
      "annotations": <object>,
      "startsAt": "<rfc3339>",
      "endsAt": "<rfc3339>",
      "generatorURL": <string>       // identifies the entity that caused the alert
    },
    ...
  ]
}

why not just move status into labels?

[metadataengine]

Hi, any update on this?
Can alertmanager developers leave the decisions on the users? That is what any configuration is there for.

[OlGe404]

Late to the party, but I'm using this config for our OpenShift alertmanager pipeline:

route:
  group_by:
    - alertname
    - severity
    - status

Hope this helps someone trying to google for this.

[gotjosh]

For @OlGe404:
I'm not sure I follow what grouping by status gives you in this context. The status itself needs to be sent from Prometheus and this is not something we do.

For everything else on the thread:

While I understand the words behind the use case, I'm not sure I follow what kind of real-world applications exist to get separate firing and resolve notifications.

The goal of the alertmanager is to be as efficient as possible in the way of sending notifications. A part of that is due to how groups work. However, indeed there is currently no way to divide firing and resolve alerts within a group in the case where you'd like to get them separate.

Before we delve into solutions, could you elaborate on what is your use case for the separation to begin with?