export ValidateMatcher for DI by kylebrandt · Pull Request #2694 · prometheus/alertmanager

kylebrandt · 2021-09-07T14:07:31Z

so third parties, Grafana in particular, can over ride the validation.

Grafana wants to do this because other data sources will have label keys with things like spaces, periods, or other characters - and looking for a better integration with alert manager.

goes with grafana/grafana#38629

requires prometheus/alertmanager#2694

silence/silence.go

config/config.go

roidelapluie · 2021-09-14T13:21:22Z

it is really unclear to me what direction grafana is taking. At some point, it might be better to just use a fork of alertmanager rather than adding abstraction layers over all the code...

i think that you could just work with the alertmanager objects, preprare them before passing them here.

it will be very confusing for users if grafana's "alertmanager" behaves differently than alertmanager.

roidelapluie · 2021-09-14T13:22:37Z

Especially since Alertmanager has not reached 1.x yet. If we miss some concepts or if there is room for improvements, we could discuss and possibly integrate those features directly in alertmanager :)

kylebrandt · 2021-09-14T13:29:28Z

@roidelapluie Sorry I didn't mean to make a draft PR in this repo yet (meant to use Grafana's but had to many GH tabs), I can close if it is a distraction - as this is still a bit exploratory (just let me know).

The objective with this branch is to allow Label Keys to be able to contain characters beyond the Open API / Prometheus label key restrictions. This is to make so alert manager can be used with other TSDBs (or things acting like TSDBs). Or more on point, other data sources that Grafana supports. (e.g. elastic has things like periods in the name, or other languages).

Renaming label keys to compatible was considered, but then then creating things like links in templates back to other systems doesn't work out.

So far in my exploration this seems to work if I:

Bypass label key validation
Take the matchers in as objects in the config so they do not need to be parsed (like the silence pb already seems to be).

@gotjosh is far more familiar with the alert manage side as well so can maybe add some context here.

roidelapluie · 2021-09-14T13:38:58Z

I have boostrapped a discussion on the grafana-developers mailing list.

config/config_test.go

Signed-off-by: Kyle Brandt <kyle@grafana.com>

…when group_wait is 0s Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Signed-off-by: Owen Diehl <ow.diehl@gmail.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Signed-off-by: Marcel Hoegel <github@mclhgl.de> Signed-off-by: Kyle Brandt <kyle@grafana.com>

…ementing limits on alerts. Update provider/mem/mem.go Co-authored-by: Julien Pivotto <roidelapluie@gmail.com> Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Limits are not used in standalone alertmanager. Signed-off-by: Peter Štibraný <pstibrany@gmail.com> Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Signed-off-by: Peter Štibraný <pstibrany@gmail.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

…er of alive goroutines. Signed-off-by: Peter Štibraný <pstibrany@gmail.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Fix JSON marshalling of empty Matchers. Signed-off-by: Peter Štibraný <pstibrany@gmail.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

* Implements a Grafana dashboard to the mixin. The dashboard aims to show an overview of the overall health of Alertmanager. Signed-off-by: ArthurSens <arthursens2005@gmail.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Signed-off-by: Sebastian Neuner <neuner@belwue.de> Signed-off-by: Kyle Brandt <kyle@grafana.com>

The expression alertmanager_cluster_members{job="alertmanager"}[5m]) is assumed to return one series for each alertmanager instance in the cluster. When running inside Kubernetes, alertmanager pods can get evicted and rescheduled. This can change the instance label and produce a new series for that alertmanager instance. When the same pod gets evicted several times in a row, there will be a short interval in which Prometheus will return values from both the new series and the old series. As a result, counting the number of series for the alertmanager_cluster_members metric will overestimate the number of instances in the given cluster. This commit modifies the the AlertmanagerMembersInconsistent alert to increase the for clause to 15m in order to reduce the probability of a false positive. Signed-off-by: fpetkovski <filip.petkovsky@gmail.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

This reverts commit 4c2a5f1. Signed-off-by: Tyler Reid <tyler.reid@grafana.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

…blish input to send. Check we populate a region for all requests. This reverts commit 4c2a5f1. Signed-off-by: Tyler Reid <tyler.reid@grafana.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Signed-off-by: Ben Ridley <benridley29@gmail.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> Signed-off-by: Kyle Brandt <kyle@grafana.com>

* target_match and source_match are DEPRECATED Signed-off-by: Dmitry Tolstoy <A-styler@ya.ru> Signed-off-by: Kyle Brandt <kyle@grafana.com>

* OpsGenie provides management of both alerts and incidents. * package opsgenie uses OpsGenie alert api, so let's refer to alert instead of incident in logs and doc to avoid confusion. Signed-off-by: Laurent CREPET <l.crepet@criteo.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Signed-off-by: Sylvain Rabot <sylvain@abstraction.fr> Signed-off-by: Kyle Brandt <kyle@grafana.com>

* cli: add new template render command Add a new template rendering command that allows users to test out their templates. This is especially needed because small bugs in templates do not surface until alertmanager actually tries to render them. * cli: permit passing alert data via a file Add a new parameter `--templatefile` for `amtool` so that it would be possible to pass custom alert data. Use an example `template.Data` if none has been passed to permit simple use-cases. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> Signed-off-by: Kyle Brandt <kyle@grafana.com>

This is the best we can do to make amtool support old releases. Supersedes #2634 Fix #2666 Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> Signed-off-by: Kyle Brandt <kyle@grafana.com>

* Add ability to skip TLS verification for amtool Signed-off-by: Nikita Nedvetskii <72229464+nedvna@users.noreply.github.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

* amtool: Detect version drift and warn users This change detects the alertmanager version when initiating the client. It ignores most errors since I expect amtool to fail later. If amtool is not compiled with proper version, we do not do anything either. We use MajorMinor for now as we have not reach 1.0, but we still allow the bugfix version number (Z in x.y.Z) to differ. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> * Add version check Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> Signed-off-by: Kyle Brandt <kyle@grafana.com>

* Add TLS option to gossip cluster Co-authored-by: Sharad Gaur <sharadgaur@gmail.com> Signed-off-by: Dustin Hooten <dustinhooten@gmail.com> * generate new certs that expire in 100 years Signed-off-by: Dustin Hooten <dustinhooten@gmail.com> * Fix tls_connection attributes Signed-off-by: Dustin Hooten <dustinhooten@gmail.com> * Improve error message Signed-off-by: Dustin Hooten <dustinhooten@gmail.com> * Fix tls client config docs Signed-off-by: Dustin Hooten <dustinhooten@gmail.com> * Add capacity arg to message buffer Signed-off-by: Dustin Hooten <dustinhooten@gmail.com> * fix formatting Signed-off-by: Dustin Hooten <dustinhooten@gmail.com> * Update version; add version validation Signed-off-by: Dustin Hooten <dustinhooten@gmail.com> * use lru cache for connection pool Signed-off-by: Dustin Hooten <dustinhooten@gmail.com> * lock reading from the connection Signed-off-by: Dustin Hooten <dustinhooten@gmail.com> * when extracting net.Conn from tlsConn, lock and throw away wrapper Signed-off-by: Dustin Hooten <dustinhooten@gmail.com> * Add mutex to connection pool to protect cache Signed-off-by: Dustin Hooten <dustinhooten@gmail.com> * fix linting Signed-off-by: Dustin Hooten <dustinhooten@gmail.com> Co-authored-by: Sharad Gaur <sharadgaur@gmail.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Signed-off-by: machinly <machinly@acm.org> Signed-off-by: Kyle Brandt <kyle@grafana.com>

* Enable support for custom callbacks as part of maintenance This enables support for custom Maintenance callbacks as part of the periodic maintenance of silences and notification logs. Effectively a no-op for the Alertmanager but allows downstream implementation to inject custom logic as part of it. Signed-off-by: gotjosh <josue.abreu@gmail.com> * Add tests Signed-off-by: gotjosh <josue.abreu@gmail.com> * Fix tests and remove whitespace Signed-off-by: gotjosh <josue.abreu@gmail.com> * Address review comments Signed-off-by: gotjosh <josue.abreu@gmail.com> * run go fmt Signed-off-by: gotjosh <josue.abreu@gmail.com> * Fix import ordering Signed-off-by: gotjosh <josue.abreu@gmail.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Signed-off-by: Kyle Brandt <kyle@grafana.com>

kylebrandt · 2021-09-23T14:22:47Z

replaced by #2716

so third parties, Grafana in particular, can over ride the validation. Grafana wants to do this because other data sources will have label keys with things like spaces, periods, or other characters - and looking for a better integration with alert manager. goes with grafana/grafana#38629 replaces #2694 Signed-off-by: Kyle Brandt <kyle@grafana.com>

so third parties, Grafana in particular, can over ride the validation. Grafana wants to do this because other data sources will have label keys with things like spaces, periods, or other characters - and looking for a better integration with alert manager. goes with grafana/grafana#38629 replaces prometheus#2694 Signed-off-by: Kyle Brandt <kyle@grafana.com>

so third parties, Grafana in particular, can over ride the validation. Grafana wants to do this because other data sources will have label keys with things like spaces, periods, or other characters - and looking for a better integration with alert manager. goes with grafana/grafana#38629 replaces prometheus#2694 Signed-off-by: Kyle Brandt <kyle@grafana.com> Signed-off-by: Marko Posavec <Marko.Posavec@infobip.com>

kylebrandt added a commit to grafana/grafana that referenced this pull request Sep 7, 2021

remove label matchers restriction

eaf9933

requires prometheus/alertmanager#2694

roidelapluie reviewed Sep 8, 2021

View reviewed changes

silence/silence.go Outdated Show resolved Hide resolved

kylebrandt mentioned this pull request Sep 14, 2021

Alerting: Allows more characters in label names so notifications are sent. grafana/grafana#38629

Merged

kylebrandt commented Sep 14, 2021

View reviewed changes

config/config.go Outdated Show resolved Hide resolved

kylebrandt commented Sep 16, 2021

View reviewed changes

config/config_test.go Outdated Show resolved Hide resolved

kylebrandt and others added 21 commits September 20, 2021 11:43

export ValidateMatcher for DI

2cfb325

Signed-off-by: Kyle Brandt <kyle@grafana.com>

wip: add object matchers

1add4f8

Signed-off-by: Kyle Brandt <kyle@grafana.com>

Fix race condition causing 1st alert to not be immediately delivered …

bf0e7de

…when group_wait is 0s Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Added unit test

6dee768

Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Lowered number of alert groups

9f61d48

Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

bump common to 0.24.0

2d1b92a

Signed-off-by: Owen Diehl <ow.diehl@gmail.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Add a note about Pushover ready-to-clone app

45574eb

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Fix example silence query parameter

b741626

Signed-off-by: Marcel Hoegel <github@mclhgl.de> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Don't return error from mem.Alerts.Put.

5b9017a

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Added group limit to dispatcher.

a46e8c8

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Only register limit metrics when they are used.

ada8581

Limits are not used in standalone alertmanager. Signed-off-by: Peter Štibraný <pstibrany@gmail.com> Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Extract TestGroupsWithLimits, and remove limit test from TestGroups.

e9bf8d9

Signed-off-by: Peter Štibraný <pstibrany@gmail.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Extract TestGroupsWithLimits, and remove limit test from TestGroups.

fa654fd

Signed-off-by: Peter Štibraný <pstibrany@gmail.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Fix compilation errors after rebase on master.

89e651f

Signed-off-by: Peter Štibraný <pstibrany@gmail.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Reduce number of dispatched alerts to avoid hitting the limit on numb…

2e7040f

…er of alive goroutines. Signed-off-by: Peter Štibraný <pstibrany@gmail.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Fix mashaling URL and Regex wrappers with nil values.

92329d3

Fix JSON marshalling of empty Matchers. Signed-off-by: Peter Štibraný <pstibrany@gmail.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Remove usage of assert package.

6d24db2

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

[mixins] Alertmanager Overview dashboard (#2540)

881ae2e

* Implements a Grafana dashboard to the mixin. The dashboard aims to show an overview of the overall health of Alertmanager. Signed-off-by: ArthurSens <arthursens2005@gmail.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

fixed typo 'weeekday_range' → 'weekday_range'

fe50ff6

Signed-off-by: Sebastian Neuner <neuner@belwue.de> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Tyler Reid and others added 24 commits September 20, 2021 11:43

Revert "Add sigv4 as a global config option"

476c8d3

This reverts commit 4c2a5f1. Signed-off-by: Tyler Reid <tyler.reid@grafana.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Break notify into submethods to create the session then create the pu…

b97ab2f

…blish input to send. Check we populate a region for all requests. This reverts commit 4c2a5f1. Signed-off-by: Tyler Reid <tyler.reid@grafana.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Ensure time interval comparisons are in UTC

1517751

Signed-off-by: Ben Ridley <benridley29@gmail.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Fix minor timezone typo to help debugging.

7a7fc5e

Signed-off-by: Ben Ridley <benridley29@gmail.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Fix test case not being included in mute count

775b6af

Signed-off-by: Ben Ridley <benridley29@gmail.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Fix typo in label to help debugging (again)

6894221

Signed-off-by: Ben Ridley <benridley29@gmail.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Update go dependencies & switch to go-kit/log

5b137df

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Docs: target_match and source_match are DEPRECATED (#2665)

59d8301

* target_match and source_match are DEPRECATED Signed-off-by: Dmitry Tolstoy <A-styler@ya.ru> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Fix test not waiting for cluster member to be ready

6ad682f

Signed-off-by: Sylvain Rabot <sylvain@abstraction.fr> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Fix main tests (#2670)

8308f1c

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Fix empty isEqual in amtool. (#2668)

5920f09

This is the best we can do to make amtool support old releases. Supersedes #2634 Fix #2666 Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Add ability to skip TLS verification for amtool (#2663)

6a3ae9d

* Add ability to skip TLS verification for amtool Signed-off-by: Nikita Nedvetskii <72229464+nedvna@users.noreply.github.com> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Release 0.23.0-rc.0

705d787

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Make the apostrophe uniform

0f07137

Signed-off-by: machinly <machinly@acm.org> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Fix some typo

9e031d4

Signed-off-by: machinly <machinly@acm.org> Signed-off-by: Kyle Brandt <kyle@grafana.com>

Fix some typo

ff0b2d3

Signed-off-by: machinly <machinly@acm.org> Signed-off-by: Kyle Brandt <kyle@grafana.com>

comment out groupby validation for now

f811cab

Signed-off-by: Kyle Brandt <kyle@grafana.com>

remove object matchers

4ee7f2a

Signed-off-by: Kyle Brandt <kyle@grafana.com>

undo another change and remove comment part

18778a1

Signed-off-by: Kyle Brandt <kyle@grafana.com>

kylebrandt mentioned this pull request Sep 23, 2021

Export ValidateMatcher for DI (#2) #2716

Merged

kylebrandt closed this Sep 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

export ValidateMatcher for DI#2694

export ValidateMatcher for DI#2694
kylebrandt wants to merge 80 commits intoprometheus:mainfrom
grafana:matcherValidation

kylebrandt commented Sep 7, 2021

Uh oh!

Uh oh!

Uh oh!

roidelapluie commented Sep 14, 2021

Uh oh!

roidelapluie commented Sep 14, 2021

Uh oh!

kylebrandt commented Sep 14, 2021 •

edited

Loading

Uh oh!

roidelapluie commented Sep 14, 2021

Uh oh!

Uh oh!

kylebrandt commented Sep 23, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

18 participants

Conversation

kylebrandt commented Sep 7, 2021

Uh oh!

Uh oh!

Uh oh!

roidelapluie commented Sep 14, 2021

Uh oh!

roidelapluie commented Sep 14, 2021

Uh oh!

kylebrandt commented Sep 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

roidelapluie commented Sep 14, 2021

Uh oh!

Uh oh!

kylebrandt commented Sep 23, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

18 participants

kylebrandt commented Sep 14, 2021 •

edited

Loading