gracefully handle HTTP 429 return codes (Too Many Requests)

[miabbott]

Downstream BZ - https://bugzilla.redhat.com/show_bug.cgi?id=1702519

In the above BZ, the RHCOS nodes are attempting to pull a container from quay.io via skopeo and the registry is returning HTTP 429 (Too Many Requests). skopeo reacts with:

msg="Error determining repository tags: Invalid status code returned when fetching tags list 429 (Too Many Requests)"

I think the error message is misleading since 429 is a valid code (https://tools.ietf.org/html/rfc6585#section-4), so at a minimum the message should be changed.

cc: @runcom

[vrothberg]

Hi @miabbott, thanks for opening the issue.

[...], so at a minimum the message should be changed.

How about "Error determining repository tags: too many requests"? Alternatively, we can do a simple "stop DOSing the registry" ;-)

[miabbott]

@vrothberg I think that message is a good start; I think it might be useful to include the HTTP code, too.

Do you think there is anything else that can be done in this scenario besides just error more cleanly?

[vrothberg]

Do you think there is anything else that can be done in this scenario besides just error more cleanly?

I don't know the penalty (i.e., how long the client is rejected) but if that's ~predictable we could try to play smart and retry after a certain amount of time. @mtrmac do you know the (default) penalty?

[miabbott]

The spec says:

...MAY include a Retry-After header indicating how long to wait before making a new request.

So it is unfortunately up to the registry/server if they decide to tell the client how long to back-off. Maybe retry according to Retry-After if it is present in the response?

[vrothberg]

Sounds great, thanks @miabbott !

[wking]

And probably do some sort of exponential backoff (60s, doubling the wait between attempts, give up after... 3? attempts) if there is no Retry-After.

[josephschorr]

We can add a Retry-After header too, if that helps

[mtrmac]

[...], so at a minimum the message should be changed.

How about "Error determining repository tags: too many requests"? Alternatively, we can do a simple "stop DOSing the registry" ;-)

This is not raw HTTP; the docker/distribution protocol has an integrated error reporting mechanism: https://github.com/docker/distribution/blob/master/docs/spec/api.md#on-failure-too-many-requests-1 .

We should probably call github.com/docker/distribution/registry/client.HandleErrorResponse or something similar in GetRepositoryTags, instead of just blindly checking for 200.

[mtrmac]

Do you think there is anything else that can be done in this scenario besides just error more cleanly?

I don't know the penalty (i.e., how long the client is rejected) but if that's ~predictable we could try to play smart and retry after a certain amount of time. @mtrmac do you know the (default) penalty?

It would be a bit weird to retry only on this single code path. I’m not sure whether it is wanted/desired for c/image to automatically retry at all (IIRC the kubelet, and the image builder, do retry at least image pulls already, so compounding the retries might be bad for error detection latency).

If we don’t retry, we definitely should return a recognizable error type (which the HandleErrorResponse call would do); should we also make the Retry-After header value available in the error? The conversation above suggests that it is not implemented in the relevant registry, and the spec does not describe it either (though, of course, using the HTTP-specified header would not be wrong :) ).

[mtrmac]

Is there a public registry, and an easy way to cause it to return 429 without affecting other users, available for testing possible fixes? Or an easy way to set up an instance locally?

Failing that, a HTTP headers+body dump of the response would probably be sufficient.

[thomasmckay]

We'll be adding a 'Retry-After' to quay, is there tooling that will honor it? Ideally we'd make use of this immediately with skopeo, will an issue fixed here make it into a skopeo build?

[rhatdan]

Yes we update containers/image into skopeo on each release of containers/image.

[mtrmac]

Right now, c/image makes an one-shot attempt, there is no retry logic in it, and no special handling of 429/Retry-After.

CRI-O/the kubelet and openshift/builder do ultimately do retries at a higher level, but by that time the Retry-After information is lost.

So, no, right now there isn’t tooling and there would be no “immediate” benefit.

---

(Adding some kind of support to c/image, whether retrying or just relaying the error in a machine-detectable way, would certainly make it into skopeo in a short while. OTOH the “machine-detectable” aspect of an error object is a bit compromised by a CLI.)

[twaugh]

We are running into this when using skopeo from the command line from a Python program which wants to manipulate tags in Quay.io.

Is anyone working on fixing this? Is there anything I can do to help?

[vrothberg]

I have some cycles now and can look into it.