Send subsequent notifications to a slack thread instead of channel

[alileza]

What did you do?

Currently alertmanager would send an alert & resolve alert in the same way as a new message. Although when there are multiple alerts being triggered at the same time, slack channel could become messy and hard to identify which alert is being resolve and when. We'd need to scroll up and down trying to link resolve alert with initial alert triggered.

What did you expect to see?

Resolve alert should be sent into slack thread of the initial alert, this way it would make slack channel much cleaner. And it gives us better visibility on duration on mean-time-to-recover

[simonpasquier]

While the feature request makes sense (and not only for Slack, see #3202), we have no easy way right now to implement this because in a multi-node setup, the subsequent notification might be sent by a different Alertmanager node which wouldn't know the thread id.
To circumvent the problem, it would require to add the thread id to the notification logs. If we were to go this route, it should be done in an extensible way so it can be applied to all notifiers (e.g. not specific to Slack).

cc @gotjosh

[gotjosh]

I'd not be opposed to storing state in the notification logs -- right now, the on-disk footprint of this data appears to be very low, and we probably get a profile to analyse the trends of resource usage.

As Simon said, my concern would be how the design looks like that's generic enough to be used by other receivers. Assuming it's a set of KVs we allow per receiver, how do we ensure we keep this efficient in-memory?

[Juliavol]

wasnt this supposed to be resolved with thread_ts in slack_config section?

I was under the impression that the alertmanager templates support this feature from version 0.22

[gotjosh]

What is thread_ts looking at the Slack configuration, I don't see that option. Am I missing something?

[gotjosh]

Ah, I get it - you meant the thread_ts field on the chat.PostMessage Slack API (see https://api.slack.com/methods/chat.postMessage).

Yeah, the idea would be to store the first notification ts and then have subsequent notification use that exact same ts. However, my questions of "how do we store this" remain unanswered.

[dnck]

Bumping this thread. I'd be interested in contributing to the development of this feature. @gotjosh are you working on it? Or know of anyone that is?

[eyalsolomon1]

@dnck can join if needed
@gotjosh is there a way we can push this ?

[tparreira93]

Can we leverage alert start time to define timestamp that would be used by Alertmanager?
It would be then a matter of Alertmanager using this value in a potential new thread_ts configuration.

[aldenjenkins]

I would love to see this implemented

[nicdard]

I would love to have this feature, I am also open to contribute this, but it would be my first contribution, so I would need some guidance :)

[jeckles]

would also be interested in seeing this feature added!

[akuzia]

Possibly the ability to change existing message, instead of opening a thread might be a better option?
For example Zabbix Slack integration already does so.