add per-tenant alertmanager metrics

[jtlisi]

Moved from #2116 because integration tests don't work for Grafana org repo PRs due to NOQUAY being set as an environment variable.

What this PR does:

This PR takes advantage of the util.MetricFamiliesPerUser struct to provide per-tenant Alertmanager metrics.

Which issue(s) this PR fixes:
Fixes #1631

Checklist

• Tests updated
• CHANGELOG.md updated

[jtlisi]

@pracucci I paired down the number of user metrics considerably. The ones that remain convey either essential basic information about the number of alerts/silences or help users avoid silent failures.

cortex_alertmanager_alerts -- Basic information that gives a basic understanding of alertmanager usage on a per-user basis
cortex_alertmanager_silences -- Same as above

cortex_alertmanager_notifications_total & cortex_alertmanager_notifications_failed_total -- These metrics are vital for detecting a silent failure in a users config. A user could have a valid Alertmanager config with a misconfigured integration. This could easily lead to a silent failure where a user is not receiving alerts. Having this metric on a per-user basis makes it easy to configure redundant alerts in either a separate Prometheus/Alertmanager system or using something like Prometheus/(Grafana Alerts).

I also made sure to remove the unused functions I added to metrics_helper.go, as well as add unit tests for the function I did add.

[pstibrany]

I have left some comments around metrics code.

[gouthamve]

LGTM with nit. I haven't taken a close look at the tests as I think Marco and Peter took a close look there.

@jtlisi Could you also resolve conversations that have been already implemented?

[pracucci]

@jtlisi Thanks for addressing my feedback. I'm a bit concerned about the "unpause" logic and wondering if there's any issue there. Please take a look at comments.

[pstibrany]

LGTM with minor nits.
(Btw, flaky test is now fixed on master)

[pracucci]

Approving cause my previous concern about the "unpause" is not an issue. However I left few comments which I would be glad if you could address before merging. Thanks @jtlisi !

[pracucci]

Why isn't exposed, while Pause() is? I think it should be specular to Pause() and being exposed too.

[jtlisi]

Makes sense

[pracucci]

it's > its