Retry on all 5xx errors

[parikls]

Sometimes we are observing an JSONDecodeError raised by the library.
This happens when replicate API returns 502 Bad Gateway

httplib log:

https://api.replicate.com/v1/models/beautyyuyanli/multilingual-e5-large/versions/a06276a89f1a902d5fc225a9ca32b6e8e6292b7f3b136518878da97c458e2bad

{
http.fragment: ,
http.method: GET,
http.query: ,
http.response.status_code: 502,
reason: Bad Gateway
}

IMO retryable status codes should be extended to cover all possible 5xx errors

https://github.com/replicate/replicate-python/blob/main/replicate/client.py#L203

[mattt]

Hi @parikls. It's not generally advisable to automatically retry on all 5XX responses, because most are not expected to succeed on subsequent requests. I can see the argument for 502, as it's sometimes returned for temporary problems, but I stand by our current behavior.

If you'd like different behavior, you can handle this error in a try / except block.

[parikls]

@mattt let's put a general 5xx retry question aside.

You already did a great job by integrating a retry mechanism in a lib, therefore as a library user I don't want to bother myself re-implementing retries in my own code. I want your library to be as simple as

await replicate.async_run('model-name', input=some_input)

Not sure I understand, if you are already aware about the 502 case, why that specific code is not retryable? Any specific reasons?
Even is there are so - as a library user I would expect to get more meaningful error subclassed from the ReplicateException, so my code can handle it properly and differentiate between unhandled errors and those which should be processed separately.

[mattt]

@parikls The thing about 502 errors is that they aren't a reliable indicator that retrying the request will succeed. It really depends on the specifics of how the API is being accessed (e.g. through a reverse proxy, VPN, or firewall).

I completely agree with you that a more specific error would be helpful. What we have right now is inadequate:

replicate-python/replicate/client.py

Lines 366 to 368 in a7f725c

	def _raise_for_status(resp: httpx.Response) -> None:
	if 400 <= resp.status_code < 600:
	raise ReplicateError(resp.json()["detail"])

I just opened #263, which I hope will make Replicate API errors more actionable. Please take a look and let me know what you think!

[mattt]

@parikls I just merged #263, which should resolve the JSONDecode exceptions you saw before. ReplicateError now exposes status and detail fields, which can give you more information about a problem that occurred:

import replicate
from replicate.exceptions import ReplicateError

try:
    output = replicate.run(
        "stability-ai/sdxl:39ed52f2a78e934b3ba6e2a89f5b1c712de7dfea535525255b1aa35c5565e08b",
        input={"prompt": "a photo of an astronaut riding a horse on Mars"},
    )
    print(output)
except ReplicateError as e:
    print(f"An error occurred: {e.status} - {e.detail}")

[parikls]

@mattt thank you!