Unable to to push/pull to an SSH-forwarded local registry with Docker Rootless

[GrimzEcho]

Description

Rootless Docker cannot access a private registry when the access needs to occur over a port-forwarded SSH connection. The only discovered workaround involves manually entering the rootless namespace (via nsenter), creating the SSH tunnel, then running docker push or docker pull.

Reproduce

Basic

1. On a remote machine docker run -d -p 5000:5000 --name registry registry:2
2. Install rootless docker on the host machine
3. docker pull hello-world
4. docker tag hello-world localhost:5000/hello-world
5. ssh -L 5000:localhost:5000 user@remote
6. docker push localhost:5000/hello-world

Result:

Using default tag: latest
The push refers to repository [localhost:5001/hello-world]
Get "http://localhost:5000/v2/": dial tcp [::1]:5000: connect: connection refused

Two things may be influencing this: the fact that rootless docker is running inside of a namespace network, and the fact that by default, Docker is invoking rootlesskit with the --disable-host-loopback flag.

CLI Issue

However, even when running rootlesskit with the arguments needed to enable general network access to localhost:5000, docker push still does not work.

On the host with rootless docker installed (note the absence of --disable-host-loopback) and using the gateway address of the slirp4netns network the following WORKS

rootlesskit --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --port-driver=slirp4netns --copy-up=/etc --copy-up=/run --propagation=rslave curl http://10.0.2.2:5000/v2/_catalog

Result:

{"repositories":[]}

However, running docker push still does not work

rootlesskit --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --port-driver=slirp4netns --copy-up=/etc --copy-up=/run --propagation=rslave docker push 10.0.2.2:5000/hello-world

The push refers to repository [10.0.2.2:5000/hello-world]
Get "http://10.0.2.2:5000/v2/": dial tcp 10.0.2.2:5000: connect: network is unreachable
[rootlesskit:child ] error: command [docker push 10.0.2.2:5000/hello-world] exited: exit status 1
[rootlesskit:parent] error: child exited: exit status 1

This is a different error than when running docker push directly

Expected behavior

One of the above mechanisms allows pushing/pulling to the private registry

docker version

Client: Docker Engine - Community
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:32:17 2023
 OS/Arch:           linux/amd64
 Context:           rootless

Server: Docker Engine - Community
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:32:17 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          1.1.1
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       builtin
  StateDir:         /tmp/rootlesskit702691539
 slirp4netns:
  Version:          1.1.8
  GitCommit:        unknown

docker info

Client: Docker Engine - Community
 Version:    24.0.6
 Context:    rootless
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.3
    Path:     /home/<user>/.docker/cli-plugins/docker-compose

Server:
 Containers: 5
  Running: 0
  Paused: 0
  Stopped: 5
 Images: 8
 Server Version: 24.0.6
 Storage Driver: fuse-overlayfs
 Logging Driver: json-file
 Cgroup Driver: none
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc version: v1.1.9-0-gccaecfc
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
 Kernel Version: 4.19.0-22-amd64
 Operating System: Debian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 31.43GiB
 Name: ...
 ID: c5d7058e-12b3-4363-8931-67f274a08282
 Docker Root Dir: /home/<user>/.local/share/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  localhost:5000
  localhost:5001
  10.0.2.2:5000
  10.0.2.2:5001
 Live Restore Enabled: false

Additional Info

The only sequence that does work is to use enterns to create the SSH tunnel, background it, then run docker push/pull. There are multiple issues with this approach, one of which is that tunnel will not be closed when the nsenter parent that created it dies.

nsenter -U --preserve-credentials -n -m -t $(cat $XDG_RUNTIME_DIR/docker.pid)
ssh -fN -L 5000:localhost:5000 user@remote
docker push localhost:5000/hello-world

I do not believe this is fundamentally an issue with rootlesskit, as Podman also uses rootlesskit and with rooless podman, pushing/pulling over an SSH tunnel works without any additional configuration.

Other things I've tried

• Using various port bindings on rootless kit (both via running the command directly and via EVs set in the rootless docker's systemd configuration
  • These either had no effect or resulted in an error about being unable to bind port 5000 (presumably because it was already being forwarded to SSH)

error: failed to expose port {tcp 127.0.0.1 5000 5000 10.0.2.100}: reply.Error: map[desc:bad request: add_hostfwd: slirp_add_hostfwd failed]

@AkihiroSuda Do you know of any workarounds to allow pushing/pulling to an SSH-tunneled registry in docker rootless?

[AkihiroSuda]

This should have been opened in https://github.com/moby/moby/discussions , as this does not seem relate to CLI

Do you know of any workarounds to allow pushing/pulling to an SSH-tunneled registry in docker rootless?

Using a non-loopback address may work

[AkihiroSuda]

FYI: there is a WIP experiment in nerdctl repo to allow the daemon to run in the host network namespace so that it can access the localhost ports:

• [rootless] Support detach netns containerd/nerdctl#2535

Eventually this should be available to Docker too

[GrimzEcho]

This should have been opened in https://github.com/moby/moby/discussions , as this does not seem relate to CLI

My thought was that since this specific issue is related to pushing and pulling via the CLI (running in a network namespace), it was applicable to this repo. There are other workarounds for accessing the host's network from rootless docker containers that didn't seem to have any parallels for running the Docker CLI commands themselves.

I'm happy to close and reopen this in Moby/discussions though If you'd like.

Using a non-loopback address may work

Could you expand on that a bit? Do you mean creating the SSH tunel/port-forward bind on something other than localhost so that it would be exposed to the rootless network namespace? I don't know what that would look like.

[AkihiroSuda]

Do you mean creating the SSH tunel/port-forward bind on something other than localhost so that it would be exposed to the rootless network namespace?

Yes, like 5000:192.168.123.45:5000

[GrimzEcho]

Closing this issue as there is both an acceptable workaround and a PR to add support for running rootless docker push/pull in the host namespace.