Automatic backoff/retry on `HTTP 429 Too Many Requests`

[soxofaan]

Some backends have middleware to throttle the number of concurrent requests, returning a 429 Too Many Requests if some limit is exceeded. The python client could provide automatic retry with (e.g. exponential) backoff functionality.

related to #440

[jdries]

The python requests library even supports this quite easily

from requests.adapters import HTTPAdapter, Retry
retries = Retry(total=5,read=10,other=10,status=10,
                backoff_factor=0.1,
                status_forcelist=[ 502, 503, 504,404,429],
                method_whitelist=["HEAD", "GET", "OPTIONS","POST"])
session.mount('https://', HTTPAdapter(max_retries=retries))
session.mount('http://', HTTPAdapter(max_retries=retries))

[soxofaan]

indeed, we might consider moving this helper from python driver to python client: https://github.com/Open-EO/openeo-python-driver/blob/b9f465dc276a3d0b0217d7f76a416b7cb943cd38/openeo_driver/util/http.py#L7-L31

and we apparently also have it in MultiBackendJobManager:

openeo-python-client/openeo/extra/job_management.py

Lines 148 to 167 in 95b89d5

	def _make_resilient(self, connection):
	"""Add an HTTPAdapter that retries the request if it fails.
	Retry for the following HTTP 50x statuses:
	502 Bad Gateway
	503 Service Unavailable
	504 Gateway Timeout
	"""
	status_forcelist = [502, 503, 504]
	retries = Retry(
	total=MAX_RETRIES,
	read=MAX_RETRIES,
	other=MAX_RETRIES,
	status=MAX_RETRIES,
	backoff_factor=0.1,
	status_forcelist=status_forcelist,
	allowed_methods=["HEAD", "GET", "OPTIONS", "POST"],
	)
	connection.session.mount("https://", HTTPAdapter(max_retries=retries))
	connection.session.mount("http://", HTTPAdapter(max_retries=retries))