nerdctl pull from private registry mirror is missing Authorization header

[networkhermit]

Description

I tried to use nerdctl with registry mirrors served by private harbor, but I got 401 Unauthorized error.

After some investigation, I found that when using registry mirrors, nerdctl doesn't pass the Authorization: Basic xxxx header to /service/token endpoints, causing the final 401 Unauthorized error.

Steps to reproduce the issue

1. configure containerd to use registry mirrors:

/etc/containerd/config.toml

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = "/etc/containerd/certs.d"

/etc/containerd/certs.d/gcr.io/hosts.toml

server = "https://gcr.io"

[host."https://harbor.example.com:8443/v2/gcr.io"]
  capabilities = ["pull", "resolve"]
  override_path = true

2. use nerdctl login to harbor.example.com:8443, optionally setup mitmproxy to view the api traffic

3. directly pull the image from harbor is successfully

❯ sudo HTTPS_PROXY=http://127.0.0.1:8080 nerdctl --debug-full pull harbor.example.com:8443/gcr.io/cadvisor/cadvisor:latest
DEBU[0000] verifying process skipped
DEBU[0000] Found hosts dir "/etc/containerd/certs.d"
DEBU[0000] Ignoring hosts dir "/etc/docker/certs.d"      error="stat /etc/docker/certs.d: no such file or directory"
DEBU[0000] The image will be unpacked for platform {"amd64" "linux" "" [] ""}, snapshotter "overlayfs".
DEBU[0000] fetching                                      image="harbor.example.com:8443/gcr.io/cadvisor/cadvisor:latest"
DEBU[0000] resolving                                     host="harbor.example.com:8443"
DEBU[0000] do request                                    host="harbor.example.com:8443" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.11+unknown request.method=HEAD url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest"
harbor.example.com:8443/gcr.io/cadvisor/cadvisor:latest: resolving      |--------------------------------------|
elapsed: 0.3 s                                             total:   0.0 B (0.0 B/s)
DEBU[0000] fetch response received                       host="harbor.example.com:8443" response.header.content-length=152 response.header.content-type="application/json; charset=utf-8" response.header.date="Tue, 27 Feb 2024 14:05:20 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=istio-envoy response.header.set-cookie="sid=7bc1d436bafba2a71e19ec32cfcc06b8; Path=/; HttpOnly" response.header.www-authenticate="Bearer realm=\"https://harbor.example.com:8443/service/token\",service=\"harbor-registry\",scope=\"repository:gcr.io/cadvisor/cadvisor:pull\"" response.header.x-envoy-upstream-service-time=7 response.header.x-request-id=cb37ff75-4c43-4d62-a565-2d767adbd4d8 response.status="401 Unauthorized" url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://harbor.example.com:8443/service/token\",service=\"harbor-registry\",scope=\"repository:gcr.io/cadvisor/cadvisor:pull\"" host="harbor.example.com:8443"
DEBU[0000] do request                                    host="harbor.example.com:8443" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.inde
harbor.example.com:8443/gcr.io/cadvisor/cadvisor:latest: resolving      |--------------------------------------|
elapsed: 2.0 s                                             total:   0.0 B (0.0 B/s)
DEBU[0002] fetch response received                       host="harbor.example.com:8443" response.header.content-length=1163 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Tue, 27 Feb 2024 14:05:22 GMT" response.header.docker-content-digest="sha256:ddadf3e2fd880deb4e0f3606d34a0d9da1165e3801116075d98a1901635dc9e8" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:ddadf3e2fd880deb4e0f3606d34a0d9da1165e3801116075d98a1901635dc9e8\"" response.header.server=istio-envoy response.header.set-cookie="sid=1fbb1be72a6e2fd04a4962cfd6997843; Path=/; HttpOnly" response.header.x-envoy-upstream-service-time=1660 response.header.x-request-id=6dfd0fa9-f711-4512-a26f-ed627a26ab73 response.status="200 OK" url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest"
DEBU[0002] resolved                                      desc.digest="sha256:ddadf3e2fd880deb4e0f3606d34a0d9da1165e3801116075d98a1901635dc9e8" host="harbor.example.com:8443"
DEBU[0002] fetch                                         digest="sha256:ddadf3e2fd880deb4e0f3606d34a0d9da1165e3801116075d98a1901635dc9e8" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1163
DEBU[0002] fetch                                         digest="sha256:68c29634fe49724f94ed34f18224316f776392f7a5a4014969ac5798a2ec96dc" mediatype=application/vnd.docker.container.image.v1+json size=4843
DEBU[0002] layer unpacked                                duration="221.474µs" layer="sha256:188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964"
DEBU[0002] layer unpacked                                duration="87.424µs" layer="sha256:2d4828968d6ca90e549ac054b8330c28081e02b31574f663fe8dae9c03275103"
DEBU[0002] layer unpacked                                duration="62.677µs" layer="sha256:b28a9b13dc6dccb175978a3e34f0ee5ec435e1328b9ded306724e490388585e4"
DEBU[0002] layer unpacked                                duration="92.493µs" layer="sha256:031224e6222c780a9783d9d287f6c25e37f16c800ba4bbb13e7f9c77036fb811"
harbor.example.com:8443/gcr.io/cadvisor/cadvisor:latest:                        resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:ddadf3e2fd880deb4e0f3606d34a0d9da1165e3801116075d98a1901635dc9e8: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:68c29634fe49724f94ed34f18224316f776392f7a5a4014969ac5798a2ec96dc:   done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 2.2 s                                                                    total:   0.0 B (0.0 B/s)

4. pulling the same image from registry mirrors fails as 401

❯ sudo HTTPS_PROXY=127.0.0.1:8080 nerdctl --debug-full pull gcr.io/cadvisor/cadvisor:latest
DEBU[0000] verifying process skipped
DEBU[0000] Found hosts dir "/etc/containerd/certs.d"
DEBU[0000] Ignoring hosts dir "/etc/docker/certs.d"      error="stat /etc/docker/certs.d: no such file or directory"
DEBU[0000] The image will be unpacked for platform {"amd64" "linux" "" [] ""}, snapshotter "overlayfs".
DEBU[0000] fetching                                      image="gcr.io/cadvisor/cadvisor:latest"
DEBU[0000] loading host directory                        dir=/etc/containerd/certs.d/gcr.io
DEBU[0000] resolving                                     host="harbor.example.com:8443"
DEBU[0000] do request                                    host="harbor.example.com:8443" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.11+unknown request.method=HEAD url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest?ns=gcr.io"
DEBU[0000] fetch response received                       host="harbor.example.com:8443" response.header.content-length=152 response.header.content-type="application/json; charset=utf-8" response.header.date="Tue, 27 Feb 2024 03:27:43 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=istio-envoy response.header.set-cookie="sid=63cc905e0a57374c5292b8273645d06c; Path=/; HttpOnly" response.header.www-authenticate="Bearer realm=\"https://harbor.example.com:8443/service/token\",service=\"harbor-registry\",scope=\"repository:gcr.io/cadvisor/cadvisor:pull\"" response.header.x-envoy-upstream-service-time=3 response.header.x-request-id=d0e29292-1e9e-4f15-ab65-51d37f1dd851 response.status="401 Unauthorized" url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest?ns=gcr.io"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://harbor.example.com:8443/service/token\",service=\"harbor-registry\",scope=\"repository:gcr.io/cadvisor/cadvisor:pull\"" host="harbor.example.com:8443"
DEBU[0000] do request                                    host="harbor.example.com:8443" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.11+unknown request.method=HEAD url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest?ns=gcr.io"
DEBU[0000] fetch response received                       host="harbor.example.com:8443" response.header.content-length=198 response.header.content-type="application/json; charset=utf-8" response.header.date="Tue, 27 Feb 2024 03:27:43 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=istio-envoy response.header.set-cookie="sid=5ce41f88f2ba4e9246430cfa8e46312f; Path=/; HttpOnly" response.header.www-authenticate="Basic realm=\"harbor\"" response.header.x-envoy-upstream-service-time=1 response.header.x-request-id=0c1dc612-bbaa-4638-ba67-18f0a69006cf response.status="401 Unauthorized" url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest?ns=gcr.io"
DEBU[0000] Unauthorized                                  header="Basic realm=\"harbor\"" host="harbor.example.com:8443"
DEBU[0000] resolving                                     host=gcr.io
DEBU[0000] do request                                    host=gcr.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.11+unknown request.method=HEAD url="https://gcr.io/v2/cadvisor/cadvisor/manifests/latest"
gcr.io/cadvisor/cadvisor:latest: resolving      |--------------------------------------|
elapsed: 133.1s                  total:   0.0 B (0.0 B/s)
INFO[0133] trying next host                              error="failed to do request: Head \"https://gcr.io/v2/cadvisor/cadvisor/manifests/latest\": Bad Gateway" host=gcr.io
FATA[0133] failed to resolve reference "gcr.io/cadvisor/cadvisor:latest": unexpected status from HEAD request to https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest?ns=gcr.io: 401 Unauthorized

5. Browse through the mitmproxy api requests, I can see the request tohttps://harbor.example.com:8443/service/token?scope=repository%3Acadvisor%2Fcadvisor%3Apull&scope=repository%3Agcr.io%2Fcadvisor%2Fcadvisor%3Apull&service=harbor-registry lacks the Authorization: Basic xxxx header, hence results the final 401 error.

I also checked the mitmproxy api traffic using ctr and crictl, on successfully image pulling from registry mirrors, the above Authorization: Basic xxxx header is both present.

Describe the results you received and expected

nerdctl pull from private registry mirror works, the credential of the mirror registry can be passed and reused from nerdctl login credential and hence adding the missing Authorization: Basic xxxx header.

What version of nerdctl are you using?

Client:
Version: 1.7.2
OS/Arch: linux/amd64
Git commit: e32c4b0
buildctl:
Version:

Server:
containerd:
Version: v1.7.13
GitCommit: 7c3aca7a610df76212171d200ca3811ff6096eb8.m
runc:
Version: 1.1.12

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

Client:
Namespace: default
Debug Mode: false

Server:
Server Version: v1.7.13
Storage Driver: overlayfs
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Log: fluentd journald json-file syslog
Storage: native overlayfs
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.7.6-arch1-1
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 125GiB
Name: arch
ID: 36b0778a-4ce8-4dca-af98-5afa21440138

[apostasie]

Hey @networkhermit !

Been thinking about this a lot.

I think the problem with this is expectations of what an http proxy does vs. what harbor does vs. what docker "content mirror" do vs. the containerd concept of "mirror".

Please take what I say with a grain of salt, as I have not been involved in this in a long time, but I think not passing credentials to mirrors is on purpose and is - or should be - part of the spec.

Nerdctl behavior is clearly intentional here, as the credentials helpers does compare the requested locator (the registry host), to the actual domain of the endpoint being contacted, and only send credentials if they match.

Most importantly, the key question should be: what happens if a "mirror" is used for two distinct registries (say, docker.io and quay.io)? Which credentials should be sent? Based on what criteria? Also, containerd "mirror" aka host does allow fine grain client TLS cert per-host. How do you achieve the same thing for credentials if you are going to pass along the credentials of the namespace?

I am not sure what ctr / crictr do or why - but this seems to me like a dangerous idea from a security perspective - besides the key unresolved questions just above - given the way authentication is done in containerd, token auth could be trivially undone to obtain the actual user credentials instead of a signed token.

I do appreciate the use-case though, and certainly do not want to brush it away.

In that context, would love to hear your thoughts on above, and also a little bit more about your use case: how many distinct users would access your mirror? Do you expect the mirror to pass along the credentials to the upstream? What should happen if the upstream is unavailable? How would that work with token auth?

Thanks!

[networkhermit]

I use k8s/k3s more than nerdctl, and my expectation naturally aligns with ctr/crictl and k8s.

Some of my previous references:

Using Mirrored Images
->
Mirroring With Containerd
->
Registry Configuration
->
Configure Registry Credentials

what happens if a "mirror" is used for two distinct registries (say, docker.io and quay.io)? Which credentials should be sent?

Actually my k3s/k8s clusters already use a shared harbor instance with multiple upstream registries, the credentials should be the one with the harbor registry itself. But I have to admit that I only use the proxy cache for public upstreams so might miss some corner cases.

Do you expect the mirror to pass along the credentials to the upstream?

No, and I don't think harbor or other proxy cache supports this mechanism.

What should happen if the upstream is unavailable?

tried in order and might have fallback, basically I think the credentials should be match with the host you currently tried

How would that work with token auth?

I don't know this usage.

I think the problem with this is expectations of what an http proxy does vs. what harbor does vs. what docker "content mirror" do vs. the containerd concept of "mirror".

I don't know much about docker mirror as it only supports dockerhub. But I think harbor or others proxy cache is more like a secondary image cache with it's own access control, other than a pass through proxy like http_proxy.

P.S. Is it possible to use registry.configs.*.auth?

[apostasie]

Thanks a lot @networkhermit

let me read this again tonight and do some tests with harbor.

At the very least we need some guidance / docs on this and probably some changes

Thanks!