Ruler doesn't report success/failure of rule evaluation

[jml]

This is partly due to the underlying Prometheus library API. We should at least suggest some upstream changes.

Having this information would make debugging issues in a running cortex deployment much easier.

[leth]

Prometheus does expose a metric: prometheus_rule_evaluation_failures_total though this doesn't allow us to do any special error handling.

[jml]

This would naturally fall out of #310 (comment)

Probably we can just put the prometheus metrics in our dashboards, making this not a Cortex bug.

[jml]

Would love to do this, but

1. am obviously not doing it
2. less likely to get to this with recent changes