Log per-query statistics

[tomwilkie]

Log:

• the query someone ran
• how long it took
• if it errored
• (perhaps) how many chunks it fetched, for s3/memcache, how many dynamodb reads etc.

[tomwilkie]

Extending this idea further - want to create a QueryStats type that gets passed around in the context, then a various places around the code base I'll increment counters in there:

• when ever we filter a chunk after fetching it
• number of chunk fetches
• number of dynamodb queries

Then we can export this object as prometheus metrics and to BigQuery for deeper analysis.

[tomwilkie]

Solving this upstream prometheus/prometheus#2554 prometheus/prometheus#1315

[bboreham]

I noticed there is a ?stats=1 option on Prometheus query URLs, which gives some timings.
I couldn't see an easy way to extend that facility, but perhaps we could add a cortexstats ?

[tomwilkie]

If you set the querier logs to debug, you get all this (and more) - and its in the traces too.