Make queries involving no metric name work

[tomwilkie]

Currently queries involving no metric name (such as count({__name__=~".+"}) by (__name__)) almost always fail with some think like:

Error: RequestError: send request failed caused by: Get https://weaveworks-dev-chunks.s3.amazonaws.com/2/1a9d5d42bc35a818%3A15bf13d7a4b%3A15bf16ca09b%3Afa6f6c42: dial tcp 52.216.0.88:443: connect: cannot assign requested address

I suspect this is because we're trying to fetch too many chunks simultaneously, and we're running out of client ports.

Also, the v7 schema isn't rolled out to prod yet.

And they are hella slow - but lets get them working first!

[aaron7]

https://docs.google.com/a/weave.works/document/d/1dIpy2XAB9seML5JjCGyauBPBy_86dig9Vef1k8rpBOM/edit?usp=sharing

[bboreham]

Current status: it's rolled out but unusably slow

[tomwilkie]

There was a ‘fix’ in Prometheus, but it turns out it was incorrect. Whole thing needs a rethink.

…

On Mon, 5 Mar 2018 at 17:28, Bryan Boreham ***@***.***> wrote: Current status: it's rolled out but unusably slow — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#416 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAbGhe-VpQo86MUEdF5ncvYziDP_qDbUks5tbXW_gaJpZM4NWZJ5> .

[bboreham]

The reason I say it's unusably slow is the entire row has to be read for an instance in before you can do anything. A small demo like Sock Shop has ~100K entries in the row, at 2-3K each that's a 200-300MB read, that has to be done serially at whatever rate the back-end will support.

Your proposed change to Prometheus was looking at a different aspect of the problem.

[bboreham]

Rolled back from v8 schema to v6 today; write throughput seems much better.
cc @khaines

[bboreham]

Tried the query count({__name__=~".+"}) by (__name__) on Weave Cloud today; got:

Error: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (215173530 vs. 67108864)

On a very small instance I do get results.

[tomwilkie]

#958 deletes the schemas that support this.

With the schema store, there are 12x less index entries. We might be able to consider removing metric name from the hash key, and having a proper __name__ index row, allowing us to do queries withing metric names. This would get hit on every write, so we'd need #607 to not hot spot, and it would be included in every read, so we'd need #947 to not hotspot there.

Even then, with millions of series, this row might be too big...