`docker manifest` `unsupported registry version` with OCI registry

[mxmeinhold]

I'm running into issues with docker manifest erroring with failed to configure transport: unsupported registry version against a https://github.com/distribution/distribution based registry since we enabled authentication.

I think it's arising from the following, since it doesn't look like we're returning that version header, though I'm not sure if it's running into other errors or not

cli/vendor/github.com/docker/docker/registry/auth.go

Lines 227 to 237 in a32cd16

	versions := auth.APIVersions(resp, DefaultRegistryVersionHeader)
	for _, pingVersion := range versions {
	if pingVersion == v2Version {
	// The version header indicates we're definitely
	// talking to a v2 registry. So don't allow future
	// fallbacks to the v1 protocol.
	foundV2 = true
	break
	}
	}

I'm not running into issues with other docker commands (docker pull and docker push have been fine), and most of my customers haven't had issues, so it seems to be limited to just docker manifest.

Attempting a workaround by injecting the header, and I understand this is flagged as experimental, but figured I'd drop an issue.

docker version

Client: Docker Engine - Community
 Cloud integration: 1.0.12
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        55c4c88
 Built:             Tue Mar  2 20:13:00 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true
Server: Docker Engine - Community

Engine:

Version:          20.10.5

API version:      1.41 (minimum version 1.12)

Go version:       go1.13.15

Git commit:       363e9a8

Built:            Tue Mar  2 20:15:47 2021

OS/Arch:          linux/amd64

Experimental:     false

containerd:

Version:          1.4.4

GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e

runc:

Version:          1.0.0-rc93

GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec

docker-init:

Version:          0.19.0

GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc., v0.6.0)
Server:

Containers: 0

Running: 0

Paused: 0

Stopped: 0

Images: 1

Server Version: 20.10.5

Storage Driver: overlay2

Backing Filesystem: extfs

Supports d_type: true

Native Overlay Diff: true

Logging Driver: json-file

Cgroup Driver: cgroupfs

Cgroup Version: 1

Plugins:

Volume: local

Network: bridge host ipvlan macvlan null overlay

Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog

Swarm: inactive

Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc

Default Runtime: runc

Init Binary: docker-init

containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e

runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec

init version: de40ad0

Security Options:

seccomp

Profile: default

Kernel Version: 5.10.25-linuxkit

Operating System: Docker Desktop

OSType: linux

Architecture: x86_64

CPUs: 4

Total Memory: 1.941GiB

Name: docker-desktop

ID: 3STY:Q4GD:VS6G:3EC2:2J3D:XJBP:I26R:MP3T:D46U:3IO2:DAVD:7GX7

Docker Root Dir: /var/lib/docker

Debug Mode: false

HTTP Proxy: http.docker.internal:3128

HTTPS Proxy: http.docker.internal:3128

Registry: https://index.docker.io/v1/

Labels:

Experimental: false

Insecure Registries:

artifact-registry-proxy.devenv-registry-mxmeinhold.ingress.general1.us1.ddbuild.io

127.0.0.0/8

Live Restore Enabled: false

[thaJeztah]

Yes, I think it should likely be ok to remove the fallbacks to v1 (it's removed in various places already). It looks like it's checking for the Docker-Distribution-Api-Version header https://github.com/moby/moby/blob/7b9275c0da707b030e62c96b679a976f31f929d3/registry/config.go#L34

And, looking at the distribution-spec; https://github.com/opencontainers/distribution-spec/blob/94fb011af1ae28dff3ae842f711d2167c5e95491/spec.md#legacy-docker-support-http-headers

Because of the origins this specification, the client MAY encounter Docker-specific headers, such as Docker-Content-Digest, or Docker-Distribution-API-Version. These headers are OPTIONAL and clients SHOULD NOT depend on them.

That said (haven't looked at that code "in context", but I see this is part of the code we "vendor" from the https://github.com/docker/docker repository, so this would require changes to be made there; I think most of the v1 fallback code should already be removed though.

[thaJeztah]

Oh! actually; looks like it's already gone on master;

cli/vendor/github.com/docker/docker/registry/auth.go

Lines 227 to 237 in a32cd16

	versions := auth.APIVersions(resp, DefaultRegistryVersionHeader)
	for _, pingVersion := range versions {
	if pingVersion == v2Version {
	// The version header indicates we're definitely
	// talking to a v2 registry. So don't allow future
	// fallbacks to the v1 protocol.
	foundV2 = true
	break
	}
	}

Code was removed in upstream moby/moby@7a50fe8#diff-75aa7bd2d0fe27421686fc58790fbe2106be87236a2cda1a9e4c9fdd700f4fc0L227-L238 in this PR: moby/moby#41757

And brought into this repository through #2835, which updated the master branch here to the latest code from the upstream repository, so this should be fixed in the upcoming 21.xx release, but you can build a version of the CLI from source

[thaJeztah]

Based on the above, I think this should be fixed on master, so let me go ahead and close this ticket, but feel free to continue the conversation

[thaJeztah]

opened a PR to remove the (now unused) const as well; moby/moby#42567