Add Jira notifier

[alin-amana]

This PR is heavily based on #601 (with ideas from github.com/fabxc/jiralerts), addressing a number of the concerns raised in #601.

For one, it doesn't open a new issue every group_interval. It sets a label of the form ALERT{alertname="FooTooHigh",job="bar"} (using the group key labels) on each issue it creates and, if found, it will reopen that issue if it's closed (by transitioning to a configurable reopen_state) unless the resolution is Won't Fix (or some other, configurable value).

The basic ideas behind it are:

1. Flapping alerts should cause as little noise as possible. As long as the issue is open, there are no automated updates to it.
2. You don't want issues to resolve automatically when the alert stops firing. Someone needs to look at them and explicitly resolve them (and hopefully either fix the issue or tweak the alert threshold in the process).
3. If the issue is simply dismissed and no action is taken, a new alert will reopen it.
4. But optionally (and disabled by default, because it may swallow notifications unintentionally) you can define a "Won't Fix" resolution and any issues closed with that resolution will never be automatically reopened.
5. All fields of an automatically generated issue may be freely changed as long as the ALERT{...} label is kept in place. And if it disappears for any reason then you simply get another issue.

In the same way as #601, it allows custom fields to be set on creation and all of them are treated as templates. Tests are on the way.

[alin-amana]

Can't seem to get the bindata to match whatever CircleCI ends up generating.

[stuartnelson3]

Don't worry about it, we can fix that part later.

@fabxc @brian-brazil thoughts? We've been advising users to create their own integrations via webhooks for a while, but jira is something that I think would have much wider adoption than those. Is this something you all want to consider?

[brian-brazil]

You're the maintainer, but I believe all new acceptances have been stopped and the concerns in #601 remain.

[alin-amana]

If I might venture an opinion: I don't mind owners/maintainers deciding they don't want a feature, it's a completely reasonable response.

But, assuming there exists a corresponding feature request, it would be nice to close it when the decision is made, with a closing comment clearly stating why, Or, in situations like #666, renamed to "Document that alerts can inhibit themselves". It would save a lot of effort and frustration on everyone's part.

[stuartnelson3]

As you've seen, we're not actively adding any new notifier types.

This is a fairly involved integration due to jira's extreme configurability. Given the widespread use of jira, though, I would definitely be interested in seeing this as a webhook integration that we can have folks try and see if it fits their needs. If you're interested, I would like to see how these configuration options you're proposing work out for users as a webhook, and if the general consensus is that it does what people want it to do, re-evaluate it for inclusion as a first class notifier.

On a somewhat related note, I'm wondering how the other maintainers feel about adding a webhooks page on http://prometheus.io, like what we have for exporters: https://prometheus.io/docs/instrumenting/exporters/

[stuartnelson3]

this should be type Secret

[brian-brazil]

On a somewhat related note, I'm wondering how the other maintainers feel about adding a webhooks page on http://prometheus.io, like what we have for exporters:

I recently added https://prometheus.io/docs/operating/integrations/

[alin-amana]

Great, I'll rework this into a webhook then. I'll be on vacation for the following 2 weeks, though, and I'm not sure how bored I'll be so it may take a while. But I will do it eventually, because we do need it over here.

[alin-amana]

So. I finally got around to doing it and here it is: https://github.com/alin-sinpalean/jiralert

Apologies to @fabxc for ripping off the name of his Jira webhook. I dropped an "s" so there's no chance of confusion. :o)

Should I add the link to https://github.com/prometheus/docs/blob/master/content/docs/operating/integrations.md ? And if so, should it replace @fabxc's jiralerts?

[brian-brazil]

We try to avoid duplication in this list, so as yours is a fully fledged solution compared to the existing example I'd say you should replace it.

I'd also suggest grabbing a port number from the exporter list.

[alin-amana]

The functionality in this PR has now been released as a fully separate webhook receiver, referenced from the Prometheus documentatoin. Thanks for the feedback and support.