Retry when safe

[laurentsenta]

Add a retry policy to idempotent requests (GET, DELETE, POST that sets data),

• default max_retries to 3 so that users automatically benefit from this change.
• Catches server errors: 429 (too many requests), 503 (service unavailable), 504 (timeout) and BadStatusLine errors (remote disconnected)

Add dependency to responses library for testing (mock the requests library).

[immerrr]

Can this conditional run into an updated urllib3 (or httplib) that got fixed with Python 3.5 ? Do we need to handle RemoteDisconnected error, too?

[laurentsenta]

RemoteDisconnected inherits from BadStatusLine their should be no difference,
but actually, after testing the code in python3 it appears that it triggers a different structure for the same error. Catching it would look like:

                isinstance(err, ConnectionError)
                and isinstance(err.args[0], ProtocolError)
                and err.args[0].args[0] == 'Connection aborted.'
                and isinstance(err.args[0].args[1], BadStatusLine))

This is terrible, I'm looking at an idiomatic way to check the list of chained exception, something like:

if BadStatusLineError in all_chained_errors(err)

Thanks for the note!

[dangra]

For a global retry code list, I think we should add 408 too.
More details at http://blog.haproxy.com/2014/05/26/haproxy-and-http-errors-408-in-chrome/

We have experienced 408's before ( #1 and #2)

[laurentsenta]

👍