Clicking 'silence alert' buttons on Alerts page does nothing

[swsnider]

It looks like the edit silence form isn't present on that page, and therefore can't be reset.

Console stack trace:

Uncaught TypeError: Cannot read property 'reset' of undefined
alerts.js:118 initNewSilence
alerts.js:156 (anonymous function)
jquery.min.js:5 x.event.dispatch
jquery.min.js:5 y.handle

[juliusv]

Thanks reporting this! This stumped me for a while, especially since I couldn't reproduce it locally at first.

Turns out, this bug happens when alerts with missing JSON fields are sent to alertmanager. The alerts HTML template contains sections like <a href="{{.Alert.Payload.GeneratorURL}}">. The GeneratorURL is actually a key in a map[string]string which is populated from the alert payload data sent to the alertmanager via JSON over HTTP. Currently, alertmanager assumes that all alerts must set correct GeneratorURL and AlertingRule properties in this payload hash. When this doesn't happen, the Go templating engine crashes during render when accessing a missing JSON field in this manner:

E0110 20:53:03.365282   12195 web.go:119] Error executing template: template: _base.html:49:76: executing "content" at <.Alert.Payload.Gener...>: invalid value; expected string

(set -logtostderr to see logging lines on stderr)

Of course, sending corrupt/incomplete alert messages to alertmanager should result in the alerts not being accepted (sender gets an HTTP error) instead of the Alerts page not rendering correctly anymore.

I'll look into fixing that!

For now, I guess you must be using some outdated version of Prometheus (or manual way) to send alerts to alertmanager? The JSON field names that Prometheus sends changed in October: prometheus/prometheus@6eecee5.

[juliusv]

For reference, this is a working example alert JSON containing one sample alert:

[
  {
    "Summary": "Alert1",
    "Description": "Alert1 is firing",
    "Labels": {
      "alertname": "alert1",
      "service": "foo-service",
      "job": "foo-job"
    },
    "Payload": {
      "Value": "1.2345",
      "GeneratorURL": "http://link-to-alerting-rule/",
      "AlertingRule": "...the firing alerting rule in text form..."
    }
  }
]

Save this to an alerts.json file, then send it to your alertmanager like this:

curl -X POST -d @alerts.json http://localhost:9090/api/alerts

[juliusv]

And I correct myself: instead of rejecting alerts with missing Payload properties, all Payload properties should be considered as optional, since they are not strictly required by anything else than the template.

[juliusv]

Ah. Actually, the only problem with non-existing values in templates arises when they are passed to functions like {{(truncate .Alert.Payload.GeneratorUrl 40)}}, where truncate is a function which expects a string. I'll wrap a conditional around that.

[swsnider]

Yeah, I was just testing the system by manually POSTing an alert in, since I hadn't even successfully compiled Prometheus yet (I'm writing a bug up about the Docker build).

[juliusv]

@swsnider Ah, does a simple non-Docker make build work for you though?

The Docker build is contributed by our friend at Docker, @discordianfish

[swsnider]

Yes, it does, but I'm using a Mac, and need to build for Linux, and cross compilation is difficult otherwise because procfs requires cgo :(

[juliusv]

Aha! Yes, a majority of core contributors is completely on Linux even on their laptops, so any help in that area is greatly appreciated :)