Stop logging details of /ready responses

[bboreham]

If the cluster is not healthy, we start to see this in the logs, which is distracting:

time="2018-04-24T14:06:43Z" level=warning msg="GET /ready (500) 4.112382ms"
time="2018-04-24T14:06:43Z" level=warning msg="Is websocket request: false\nGET /ready HTTP/1.1\r\nHost: 10.244.204.176:80\r\nConnection: close\r\nAccept-Encoding: gzip\r\nConnection: close\r\nUser-Agent: kube-probe/1.9\r\n\r\n"
time="2018-04-24T14:06:43Z" level=warning msg="Response: "

Possibly the code for this is in weaveworks/common

[juliusv]

The logging code is in https://github.com/weaveworks/common/blob/master/middleware/logging.go#L39-L50 and that logging middleware gets added to any server created with server.New() (which Cortex uses): https://github.com/weaveworks/common/blob/master/server/server.go#L73

And then the /ready endpoint is added on that server here: https://github.com/weaveworks/cortex/blob/master/cmd/ingester/main.go#L89

It's not clear to me how one would selectively disable logging for one endpoint on this server in a clean way. The first intuition would be to allow specifying an option when registering a route with the server, but that feels a bit like overkill. Another (hacky?) option would be to return a special header X-Foo-Dont-Log-Me from handler functions which want to be exempt from logging, and the middleware would then honor that. That is very non-invasive, but also a bit magical.

[bboreham]

Thanks for the analysis, @juliusv.

The thing is, the logging code is interpreting 500 as meaning "terrible error! log as much as possible for debugging!"

It already has a bit of a hack to exclude 502 ("bad gateway"), probably because our authenticating proxy gets those when something is offline and we know it's pointless shouting about it.

Suggestion: have Cortex /ready return 503 ("service unavailable") instead of 500, and extend the above hack.

[juliusv]

@bboreham That is too easy! :) No, sounds great. Will do.