Refactor Ruler/Alertmanager API & Decouple configdb

[jtlisi]

This PR accomplishes the following

• Add a separate API for the Ruler and Alertmanager based on this design document
• Add a GCS backend for both the alertmanager and ruler API
• Update the polling mechanism to use a KV store backed change detection system for the decouple APIs or use a generation based configdb client
• Update the scheduler to use a user-based context to allow for cancellations after updates or deletions
• Pass a prometheus register with the users ID to the prometheus rule group to get user level prom eval metrics

Any feedback or suggestions are greatly appreciated

[jtlisi]

This is moved to the scheduler with different buckets and an additional metric for missed iterations

[bboreham]

Update the polling mechanism to use a KV store backed change detection system

can you explain this a bit more? "kv" doesn't appear in the design document.

[jtlisi]

@bboreham

Sorry about that, the KV system was a recent addition since the shortcomings of managing Polls within the GCS became clear. With the recent updates to the KV package it became rather simple to implement a simple change detection system.

As implemented, the KV store change detection system wraps the ruler and alertmanager api and tracks when configs are created/edited/deleted. Then when the scheduler polls for updated rule groups the kv store will be referenced to find rules that have been updated since the previous poll.

Initially the idea was to have the underlying implementation manage change detection and polling. However, by using the KV store it reduces the complexity of a rule storage backend, which will make it trivial to add additional config storage backends (S3, Azure Blob store, PostGres, boltdb, etc...).

I'll update the design document to account for the KV store asap.

[jtlisi]

Closing this PR in favor of smaller more readable PRs