Updated handling of gRPC Context Cancelled

[nrjpoddar]

Signed-off-by: Neeraj Poddar neeraj@aspenmesh.io

[nrjpoddar]

This PR #151 was supposed to fix it but I noticed that the logging was still happening on Cancelled contexts as the error checking was incorrect.

[nrjpoddar]

@bboreham do you mind taking a look?

[bboreham]

The code looks fine, but given the previous one also looked fine, can we get any more assurance this time?
How did you find out what to do, both times? Can we point to a reference?
Is it feasible to write a test?

[nrjpoddar]

Haha, totally agree and a valid point.

How did you find out what to do, both times? Can we point to a reference? -> First time I didn't understand correctly what error type was being returned here for context cancellations. After we upgraded ingester in our environment which had my fixes I was still seeing this error:

ingester level=warn ts=2019-06-21T15:54:23.748137807Z caller=grpc_logging.go:54 method=/cortex.Ingester/QueryStream duration=59.418013ms err="rpc error: code = Canceled desc = context canceled" msg="gRPC\n"

From this I derived that the error was a gRPC error type.

Unit test for this might be a bit tricky but I think I can write it with sufficient dependency injection.

[bboreham]

Having extended the tests to cover cancellation, I believe this PR was incorrect - we need to cover both context.Canceled and grpc/codes.Canceled. I created #180 to replace.