Add etcd notifier

[dtnaylor]

Add notifier that puts/deletes keys in an etcd KV store using etcd's grpc-gateway.

I’m not sure how much demand for this there is, so if you don’t think it’s worth supporting, then consider this a +1 for #701.

A sample configuration:

# sample etcd notifier config that writes a KV pair when an alert fires
# and deletes it when the alert is resolved
etcd_configs:
 - firing:
     url: http://localhost:2379/v3alpha/kv/put
     key_annotation: 'etcd_key'
     value_annotation: 'etcd_value'
   resolved:
     url: http://localhost:2379/v3alpha/kv/deleterange
     key_annotation: 'etcd_key'
     value_annotation: 'etcd_value'

And a sample rule:

ALERT QueueFull
  IF queue_occupancy > 0.9
  FOR 5s
  LABELS { severity = “warning” }
  ANNOTATIONS {
    summary = “…”,
    etcd_key = "/queue_occupancy/{{ $labels.queue_name }}",
    etcd_value = "{{ $value }}",
  }

[stuartnelson3]

Thanks for the pr! As you noticed, we're not actively adding notifiers at the moment as we try to figure out something more flexible. For the time being, we recommend implementing custom notifiers via the webhook notifier.

[dtnaylor]

Great, we'll stay tuned to see what you come up with :)

In case it helps your planning, the two things we need in this case that the webhook notifier doesn't (currently) offer are (1) the ability to add custom keys to the POSTed JSON and (2) the ability to base64 encode the values (maybe base64encode could be added as a template function for Prometheus alerts?).