Permit HTTP redirects globally and add RFC references

[tianon]

• allow registries to redirect any request per RFC 9110 (section 15.4); status codes in this spec are those returned after redirects have been followed; clients MUST NOT forward Authorization headers across host boundaries

  • Closes Pulling a blob doesn't specify 307 as a valid response status code #397

• add 429 / Retry-After guidance per RFC 6585 (section 4) and RFC 9110 (section 10.2.3)

• note that RFC 9110 governs HTTP behaviors not otherwise specified

  • Closes Support pull/push redirect #299

• normalize all RFC citations to consistent rfc-editor.org URLs with section numbers in link text

See also:

• Add 307 as valid response for pulling blobs #398 (prior attempt at the 307 fix, closed without merging)
• spec: add missing TOOMANYREQUESTS (429) error-code #209 (added TOOMANYREQUESTS to the error table; this adds the RFC citations)
• Historical curiosity: Why did we delete half the spec, again? #443
• spec is missing details #446

[tianon]

Implementation evidence:

307:

• distribution v2.7.1 registry/storage/blobserver.go#L41 (server)
• cue-labs/oci ociregistry/ociserver/reader.go#L61 (server)
• containerd core/remotes/docker/resolver.go#L647 (client)
• regclient internal/reghttp/http.go#L803 (client)

429 / Retry-After:

• olareg olareg.go#L184 (server, emits Retry-After)
• regclient internal/reghttp/http.go#L663 (client, reads Retry-After)

[sudo-bmitch]

LGTM.

[mikebrow]

Suggested change

	Registries MAY respond to any request with a redirect per [RFC 9110 (section 15.4)](https://www.rfc-editor.org/rfc/rfc9110#section-15.4); clients MUST follow such redirects, and MUST NOT forward `Authorization` headers across host boundaries unless explicitly configured to do so.
	Registries MAY respond to any request with a redirect per [RFC 9110 (section 15.4)](https://www.rfc-editor.org/rfc/rfc9110#section-15.4); clients following such redirects MUST NOT forward `Authorization` headers across host boundaries unless explicitly configured to do so.

Don't think we should force clients to accept redirects, that feels like an optional consideration.

[tianon]

It's not "forcing", it's documenting reality.

Do you have a counter example of a client that doesn't follow redirects today? Any such client won't work against almost every single major registry, which is really compelling evidence for "this should be normative in the spec"

[mikebrow]

keyword .. "should be normative" not MUST follow.. should be noted that in some circles redirects are not normative.

Not aware of any clients that disallow redirects outside some restrictions on the number of redirects.

[mikebrow]

https://github.com/containerd/containerd/blob/main/core/remotes/docker/resolver.go#L649-L651

[tianon]

Limiting the number of redirects followed is totally reasonable and actually called out in the RFC itself:

A client SHOULD detect and intervene in cyclical redirections (i.e., "infinite" redirection loops).

Note: An earlier version of this specification recommended a maximum of five redirections ([RFC2068], Section 10.3). Content developers need to be aware that some clients might implement such a fixed limitation.

I'd be willing to soften to a SHOULD from a MUST, but removing the strong language is (IMO) not being honest about how this actually works in practice -- for example, any client that doesn't follow redirects at all will literally not work with any of the largest public registries. That's stronger ecosystem evidence than would be covered by just a recommendation.

[mikebrow]

Limiting the number of redirects followed is totally reasonable and actually called out in the RFC itself:

A client SHOULD detect and intervene in cyclical redirections (i.e., "infinite" redirection loops).

Note: An earlier version of this specification recommended a maximum of five redirections ([RFC2068], Section 10.3). Content developers need to be aware that some clients might implement such a fixed limitation.

I'd be willing to soften to a SHOULD from a MUST, but removing the strong language is (IMO) not being honest about how this actually works in practice -- for example, any client that doesn't follow redirects at all will literally not work with any of the largest public registries. That's stronger ecosystem evidence than would be covered by just a recommendation.

soften to SHOULD language SGTM

I think it would also be fair to mention the need for reasoned following of redirects to enable working with the preponderance of public registries. Another reasonable response might be to check redirect allowance config for certain regions / providers, or just to include an info/debug log of the redirects so as not to hide them from the user.. etc..

[jcarter3]

Approaching this from a different perspective: A valid client is expected to work with a valid registry.

In order to accomplish this, a client must have more strict rules than the registry. If it is valid for a registry to return a redirect, then a client must handle that. If it does not, then the client does not work with valid registries, and thus is not a valid client.

If a custom client is created for internal use cases that does not want to honor redirects, this is fine. But it's also not a spec compatible client. Which is also fine.

[mikebrow]

this "new" spec language is based on existing behavior typically considered by at least some secure guidelines as a red flag... Obviously, redirects can lead to hijacking exploits. It's fair to say the client MUST follow redirects where the desire is to load from the vast majority, or what not, of registries. Simple enough to build a registry that does not redirect, or if it does, keeps the redirect to some reasonable secure domain and count of, e.g. local registry and/or domain cached registry, ... IMO the client SHOULD be allowed to reject redirects under certain conditions and remain in spec.