Use Prometheus 2.0

[jml]

We want to start depending on the Prometheus 2.0 code base. There are a bunch of reasons:

• we want to better able to contribute stuff back upstream
• we want the latest & greatest features
  • new rules format
  • better staleness handling

The main sticking point is that Cortex's storage is pretty intimately tied with Prometheus 1.0's storage system (@tomwilkie, @juliusv, please correct me if I'm wrong), and Prometheus 2.0 completely changes this.

The protobufs all look stable. A bunch of internal Go interfaces have also changed (Querier, SeriesIterator, etc.).

I think a sensible approach would be to:

• fork Prometheus 1.0's storage package
• use that within Cortex
• switch everything else to use Prometheus 2.0

I expect that there will be many changes that are connected (the new query engine seems bound up with the new storage system, and I haven't explored what needs to happen there), but guess that the above is a broadly OK strategy.

Thoughts?

[jml]

Oh, also, I don't think we need to wait for Prometheus 2.0 to be finally released.

[tomwilkie]

+1 thats exactly what I was thinking.

We should move to tsdb at some point, but that orthogonal to this. Really this is about using the PromQL engine for the 2.0 branch.

[jml]

I should point out that most of the OP derives from a conversation between @juliusv & myself, and although I don't remember who came up with what, probably all the good ideas are his.

[juliusv]

FTR, what I'm currently doing:

• Hard-forked (copied over) storage/local and related Prometheus 1.x packages into Cortex because we'll still need to depend on that for a while.
• Changed Cortex packages to depend on that copy, make things compile.
• Updated vendoring of Prometheus to 2.0 for the rest of the packages (promql, rules, config, notifier etc.).
• Currently writing adapters from PromQL 2.0->Cortex storage and RuleGroup 2.0 -> Cortex storage and some other things that will graft everything together properly again so that the user-visible parts of Prometheus 2.0 (new rule format and PromQL bits) will work.

Actually changing Cortex's internal storage itself to be more like Prometheus 2.0's storage (or not) is both a decision and a task I'll leave up to more storage-inclined people like @tomwilkie. I see that as an orthogonal (not really user-visible) concern...

[juliusv]

I have a large blob of changes piling up in the prom2-wip branch (it kinda all needs to be done at the same time), but for now, here's one necessary upstream change to re-add contexts to the storage query interface to enable multi-tenancy with the 2.0 packages:

prometheus/prometheus#3230

[jml]

@juliusv mentioned these follow-up ideas:

1. update to latest post-2.0 Prometheus packages that add a native remote read API (done)
2. update to latest Alertmanager packages, which requires a log injection change, where it would probably make sense to do this first:
3. change Cortex codebase to use the new-style Prometheus logging instead of the old one,
4. remove old model (sample, labelset, etc.) types in deeper layers of the Cortex code base, not just where they touch Prom packages