SD:Consul - On an EOF error continue the loop

[kcajmagic]

The loop in question

What is happening

In a rare occurrence, an EOF error can be received from the consul agent and new updates will not be received. When logging into the instance, no traffic from go-kit to the consul agent is observed. I believe this is from the loop exiting. The consul agent logged an error rpc error making call: EOF and the application had the error Unexpected response code: 500 (rpc error making call: EOF)

I have been unable to consistently reproduce the issue. However, the app(go-kit) and consul seem to be happy, even though there is an inconsistency in the sd instances.

What is expected

If an EOF error is received do an exponential back like other errors

This way when an EOF error occurs and both the consul agent and the app are alive, both will continue to work as expected.

A solution (the one on the top of my head)

Have a custom error for the quitc, and use this error to exit the loop.