Empty string "--volume" causes unhelpful error messages

[dries007]

Description

When you start a docker container with an empty string passed to --volume you get unhelpful error messages.

I would like to see that this is caught by the CLI with a nice error message.

Reproduce

docker run -v "" --rm hello-world

Output:

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: open /proc/sys/net/ipv4/ip_unprivileged_port_start: no such file or directory: unknown.

Expected behavior

Something along the lines of invalid argument "" for "--volume" flag: EOF.

This would match what happens when you run docker run --mount "" --rm hello-world.

docker version

Client:
 Version:           23.0.1
 API version:       1.42
 Go version:        go1.20
 Git commit:        a5ee5b1dfc
 Built:             Sat Feb 11 13:58:04 2023
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          23.0.1
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.20
  Git commit:       bc3805a0a0
  Built:            Sat Feb 11 13:58:04 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.6.18
  GitCommit:        2456e983eb9e37e47538f59ea18f2043c9a73640.m
 runc:
  Version:          1.1.4
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.10.3
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.16.0
    Path:     /usr/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 31
  Running: 2
  Paused: 0
  Stopped: 29
 Images: 1099
 Server Version: 23.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640.m
 runc version: 
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.12-zen1-1-zen
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 31.29GiB
 Name: DriesPC
 ID: 3S7A:XOUF:QR4M:SBSF:25EG:PTGE:UDJP:H7NB:PQB6:W473:DHCH:QEH7
 Docker Root Dir: /mnt/ssd2/docker
 Debug Mode: false
 Username: dries007
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

This happened because I was using a script in which a variable was accidentally unset. This caused a very long search since it was not obvious what was the root cause.

[thaJeztah]

Thanks for reporting! Yes, looks like there's some improvements to be made 🤔

I gave your example a try both on Docker 23.0.1 and 20.10, and it looks like the same happens on both, so the "good new" is that it's not a regression.

When putting the daemon in "debug mode", I can see that the following API calls are made;

DEBU[2023-02-27T16:08:26.716504702Z] Calling POST /v1.42/containers/create
DEBU[2023-02-27T16:08:26.718090142Z] form data: {"AttachStderr":true,"AttachStdin":false,"AttachStdout":true,"Cmd":null,"Domainname":"","Entrypoint":null,"Env":null,"HostConfig":{"AutoRemove":true,"Binds":null,"BlkioDeviceReadBps":[],"BlkioDeviceReadIOps":[],"BlkioDeviceWriteBps":[],"BlkioDeviceWriteIOps":[],"BlkioWeight":0,"BlkioWeightDevice":[],"CapAdd":null,"CapDrop":null,"Cgroup":"","CgroupParent":"","CgroupnsMode":"","ConsoleSize":[60,234],"ContainerIDFile":"","CpuCount":0,"CpuPercent":0,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpuShares":0,"CpusetCpus":"","CpusetMems":"","DeviceCgroupRules":null,"DeviceRequests":null,"Devices":[],"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":null,"GroupAdd":null,"IOMaximumBandwidth":0,"IOMaximumIOps":0,"IpcMode":"","Isolation":"","Links":null,"LogConfig":{"Config":{},"Type":""},"MaskedPaths":null,"Memory":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":-1,"NanoCpus":0,"NetworkMode":"default","OomKillDisable":false,"OomScoreAdj":0,"PidMode":"","PidsLimit":0,"PortBindings":{},"Privileged":false,"PublishAllPorts":false,"ReadonlyPaths":null,"ReadonlyRootfs":false,"RestartPolicy":{"MaximumRetryCount":0,"Name":"no"},"SecurityOpt":null,"ShmSize":0,"UTSMode":"","Ulimits":null,"UsernsMode":"","VolumeDriver":"","VolumesFrom":null},"Hostname":"","Image":"hello-world","Labels":{},"NetworkingConfig":{"EndpointsConfig":{}},"OnBuild":null,"OpenStdin":false,"StdinOnce":false,"Tty":false,"User":"","Volumes":{"":{}},"WorkingDir":""}

Formatting that request, it looks like it's sending "" as volume name;

"Volumes": {
   "": {}
},

Full request below;

Details

{
  "AttachStderr": true,
  "AttachStdin": false,
  "AttachStdout": true,
  "Cmd": null,
  "Domainname": "",
  "Entrypoint": null,
  "Env": null,
  "HostConfig": {
    "AutoRemove": true,
    "Binds": null,
    "BlkioDeviceReadBps": [],
    "BlkioDeviceReadIOps": [],
    "BlkioDeviceWriteBps": [],
    "BlkioDeviceWriteIOps": [],
    "BlkioWeight": 0,
    "BlkioWeightDevice": [],
    "CapAdd": null,
    "CapDrop": null,
    "Cgroup": "",
    "CgroupParent": "",
    "CgroupnsMode": "",
    "ConsoleSize": [60, 234],
    "ContainerIDFile": "",
    "CpuCount": 0,
    "CpuPercent": 0,
    "CpuPeriod": 0,
    "CpuQuota": 0,
    "CpuRealtimePeriod": 0,
    "CpuRealtimeRuntime": 0,
    "CpuShares": 0,
    "CpusetCpus": "",
    "CpusetMems": "",
    "DeviceCgroupRules": null,
    "DeviceRequests": null,
    "Devices": [],
    "Dns": [],
    "DnsOptions": [],
    "DnsSearch": [],
    "ExtraHosts": null,
    "GroupAdd": null,
    "IOMaximumBandwidth": 0,
    "IOMaximumIOps": 0,
    "IpcMode": "",
    "Isolation": "",
    "Links": null,
    "LogConfig": {
      "Config": {},
      "Type": ""
    },
    "MaskedPaths": null,
    "Memory": 0,
    "MemoryReservation": 0,
    "MemorySwap": 0,
    "MemorySwappiness": -1,
    "NanoCpus": 0,
    "NetworkMode": "default",
    "OomKillDisable": false,
    "OomScoreAdj": 0,
    "PidMode": "",
    "PidsLimit": 0,
    "PortBindings": {},
    "Privileged": false,
    "PublishAllPorts": false,
    "ReadonlyPaths": null,
    "ReadonlyRootfs": false,
    "RestartPolicy": {
      "MaximumRetryCount": 0,
      "Name": "no"
    },
    "SecurityOpt": null,
    "ShmSize": 0,
    "UTSMode": "",
    "Ulimits": null,
    "UsernsMode": "",
    "VolumeDriver": "",
    "VolumesFrom": null
  },
  "Hostname": "",
  "Image": "hello-world",
  "Labels": {},
  "NetworkingConfig": {
    "EndpointsConfig": {}
  },
  "OnBuild": null,
  "OpenStdin": false,
  "StdinOnce": false,
  "Tty": false,
  "User": "",
  "Volumes": {
    "": {}
  },
  "WorkingDir": ""
}

If I remove the --rm from the docker run;

docker run -v "" hello-world

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: open /proc/sys/net/ipv4/ip_unprivileged_port_start: no such file or directory: unknown.

docker ps -a
CONTAINER ID   IMAGE         COMMAND    CREATED         STATUS    PORTS     NAMES
74b26d0d723a   hello-world   "/hello"   3 minutes ago   Created             awesome_mcnulty

When inspecting that container's config;

docker container inspect awesome_mcnulty

There's also the "Volumes": {"": {}}, (as expected), but in addition, the Mounts contains an entry to mount an anonymous volume;

"Mounts": [
    {
        "Type": "volume",
        "Name": "a8a603cb41b14aa42ef9969998fe709db843f2fffec88fdf1d9b162cf9767287",
        "Source": "/var/lib/docker/volumes/a8a603cb41b14aa42ef9969998fe709db843f2fffec88fdf1d9b162cf9767287/_data",
        "Destination": ".",
        "Driver": "local",
        "Mode": "",
        "RW": true,
        "Propagation": ""
    }
],

Full inspect output:

Details

[
    {
        "Id": "74b26d0d723a120b9d5afcdbfdbbfa2c657c674d98ca24e96ff10aef53e9cc3b",
        "Created": "2023-02-27T16:21:00.860555084Z",
        "Path": "/hello",
        "Args": [],
        "State": {
            "Status": "created",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 127,
            "Error": "failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: open /proc/sys/net/ipv4/ip_unprivileged_port_start: no such file or directory: unknown",
            "StartedAt": "0001-01-01T00:00:00Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412",
        "ResolvConfPath": "/var/lib/docker/containers/74b26d0d723a120b9d5afcdbfdbbfa2c657c674d98ca24e96ff10aef53e9cc3b/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/74b26d0d723a120b9d5afcdbfdbbfa2c657c674d98ca24e96ff10aef53e9cc3b/hostname",
        "HostsPath": "/var/lib/docker/containers/74b26d0d723a120b9d5afcdbfdbbfa2c657c674d98ca24e96ff10aef53e9cc3b/hosts",
        "LogPath": "/var/lib/docker/containers/74b26d0d723a120b9d5afcdbfdbbfa2c657c674d98ca24e96ff10aef53e9cc3b/74b26d0d723a120b9d5afcdbfdbbfa2c657c674d98ca24e96ff10aef53e9cc3b-json.log",
        "Name": "/awesome_mcnulty",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                60,
                234
            ],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": [],
            "BlkioDeviceWriteBps": [],
            "BlkioDeviceReadIOps": [],
            "BlkioDeviceWriteIOps": [],
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/b2e919471d78b621b3b86fd9c6c08bc874ba6db48b820a8e625305919f5afa46-init/diff:/var/lib/docker/overlay2/f5fbe8ccb18d1f750d09eb21d904ecff10dcf44656e0e1655f2fb1f09b90506e/diff",
                "MergedDir": "/var/lib/docker/overlay2/b2e919471d78b621b3b86fd9c6c08bc874ba6db48b820a8e625305919f5afa46/merged",
                "UpperDir": "/var/lib/docker/overlay2/b2e919471d78b621b3b86fd9c6c08bc874ba6db48b820a8e625305919f5afa46/diff",
                "WorkDir": "/var/lib/docker/overlay2/b2e919471d78b621b3b86fd9c6c08bc874ba6db48b820a8e625305919f5afa46/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "a8a603cb41b14aa42ef9969998fe709db843f2fffec88fdf1d9b162cf9767287",
                "Source": "/var/lib/docker/volumes/a8a603cb41b14aa42ef9969998fe709db843f2fffec88fdf1d9b162cf9767287/_data",
                "Destination": ".",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "74b26d0d723a",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "/hello"
            ],
            "Image": "hello-world",
            "Volumes": {
                "": {}
            },
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "5258a296cff5bfc90a8fe433665473bcae77698417b2cd9aa4eed6b6598452b3",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/5258a296cff5",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "b6a34a8f06831306aef0624edacb073427d46fa866a652ef61cc998a5fe3d038",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

And looking at the list of volumes, there's indeed a volume created;

docker volume ls
DRIVER    VOLUME NAME
local     a8a603cb41b14aa42ef9969998fe709db843f2fffec88fdf1d9b162cf9767287

So what looks to be happening;

• The CLI sends a -v "" (volume with an empty name), but destination set for inside the container
• The daemon creates an anonymous volume (as no name was specified for the volume to be created)
• The daemon sets . (current directory) as mount-point for the volume
• This information gets sent to the OCI runtime (runc), which mounts the volume to . (/), effectively "masking" the container's whole rootfs (mounting an empty directory over /), which then fails (as paths such as /proc/sys/net/ipv4/ip_unprivileged_port_start are masked, and thus cannot be found inside the container's rootfs).

It looks like there's some potential other corner-cases as well; for example: "":/ is treated as a literal :/ as mount-point (which gets normalised to :);

docker run -it -v "":/ hello-world

"Volumes": {
    ":/": {}
},

"Mounts": [
    {
        "Type": "volume",
        "Name": "a164673e992aa46c8099b2bd34dcde4ed037a5b3c7290b6cb056bafbd8b99213",
        "Source": "/var/lib/docker/volumes/a164673e992aa46c8099b2bd34dcde4ed037a5b3c7290b6cb056bafbd8b99213/_data",
        "Destination": ":",
        "Driver": "local",
        "Mode": "",
        "RW": true,
        "Propagation": ""
    }
],

But using a proper name for the volume, and / as target has special handling;

docker run -it -v myvolume:/ hello-world
docker: Error response from daemon: invalid volume specification: 'myvolume:/': invalid mount config for type "volume": invalid specification: destination can't be '/'.
See 'docker run --help'.

I seem to recall I had a PR to better normalize the destination (including making sure that . also produces that error), but it wasn't accepted at the time (perhaps should be revisited).

[thaJeztah]

related;

• Validation of volumes (both during build, and run) is incomplete moby/moby#35879
• Check that the VOLUME instruction arguments are not '.' or '/' moby/moby#35981