docker image build does not RUN chmod Dockerfile directive depending on host OS

[M15terHyde]

Description

This question was originally posted in Docker Forums. Link here
After adding the line:
RUN chmod g+rwx /etc/bind
to my Dockerfile to fix a directory permissions issue the command is run successfully when building and running on a Windows 10 machine but the command does not run when built and run on a Ubuntu 20 machine. The same Dockerfile produces different results for different machines.

Steps to reproduce the issue:

1. On a Windows machine create the following Dockerfile:

FROM internetsystemsconsortium/bind9:9.16

RUN chmod g+rwx /etc/bind

2. Run:

docker image build -t mybind:latest .
docker run mybind:latest ls -la /etc/bind

3. Take note of the group permissions for directory: .

4. On a seperate Ubuntu machine create the same Dockerfile as before:

FROM internetsystemsconsortium/bind9:9.16

RUN chmod g+rwx /etc/bind

5. Run:

docker image build -t mybind:latest .
docker run mybind:latest ls -la /etc/bind

6. Notice the group permissions on . directory were not applied as the chmod command instructed.

Describe the results you received:

My results for the Windows machine:

docker image build -t mybind:latest .
# Builds fine
 => [internal] load build definition from Dockerfile                                                               0.0s
 => => transferring dockerfile: 31B                                                                                0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => [internal] load metadata for docker.io/internetsystemsconsortium/bind9:9.16                                    1.9s
 => [1/2] FROM docker.io/internetsystemsconsortium/bind9:9.16@sha256:741c12d794f1af570898d37288635366ead7d9a1ee4a  0.0s
 => CACHED [2/2] RUN chmod g+rwx /etc/bind                                                                         0.0s
 => exporting to image                                                                                             0.0s
 => => exporting layers                                                                                            0.0s
 => => writing image sha256:9cf71c8cd1ff3bab424702906965b1d597773d447820bb915fd8e19a44be44b8                       0.0s
 => => naming to docker.io/library/mybind:latest

docker run mybind:latest ls -la /etc/bind
# Results: Looks good.
total 56
drwxrwsr-x 2 root bind 4096 Sep 26 01:59 .
drwxr-xr-x 1 root root 4096 Sep 26 01:59 ..
-rw-r--r-- 1 root root 1991 Sep 16 07:55 bind.keys
-rw-r--r-- 1 root root  237 Sep 16 07:54 db.0
-rw-r--r-- 1 root root  271 Sep 16 07:54 db.127
-rw-r--r-- 1 root root  237 Sep 16 07:54 db.255
-rw-r--r-- 1 root root  353 Sep 16 07:54 db.empty
-rw-r--r-- 1 root root  270 Sep 16 07:54 db.local
-rw-r--r-- 1 root bind  463 Sep 16 07:54 named.conf
-rw-r--r-- 1 root bind  498 Sep 16 07:54 named.conf.default-zones
-rw-r--r-- 1 root bind  165 Sep 16 07:54 named.conf.local
-rw-r--r-- 1 root bind  846 Sep 16 07:54 named.conf.options
-rw-r----- 1 bind bind  100 Sep 21 19:30 rndc.key
-rw-r--r-- 1 root root 1317 Sep 16 07:54 zones.rfc1918

Ubuntu:

docker image build -t mybind:latest .
# Built fine
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM internetsystemsconsortium/bind9:9.16
 ---> 225a67715eb2
Step 2/2 : RUN chmod g+rwx /etc/bind
 ---> Running in 70f313e8dbaa
Removing intermediate container 70f313e8dbaa
 ---> 7bc872eeee26
Successfully built 7bc872eeee26
Successfully tagged mybind:latest

docker run mybind:latest ls -la /etc/bind
# Ahhhh there it is. Permissions weren't applied.
total 56
drwxr-sr-x 2 root bind 4096 Sep 26 02:09 .
drwxr-xr-x 1 root root 4096 Sep 26 02:09 ..
-rw-r--r-- 1 root root 1991 Aug 20 12:41 bind.keys
-rw-r--r-- 1 root root  237 Aug 20 12:40 db.0
-rw-r--r-- 1 root root  271 Aug 20 12:40 db.127
-rw-r--r-- 1 root root  237 Aug 20 12:40 db.255
-rw-r--r-- 1 root root  353 Aug 20 12:40 db.empty
-rw-r--r-- 1 root root  270 Aug 20 12:40 db.local
-rw-r--r-- 1 root bind  463 Aug 20 12:40 named.conf
-rw-r--r-- 1 root bind  498 Aug 20 12:40 named.conf.default-zones
-rw-r--r-- 1 root bind  165 Aug 20 12:40 named.conf.local
-rw-r--r-- 1 root bind  846 Aug 20 12:40 named.conf.options
-rw-r----- 1 bind bind  100 Aug 25 14:43 rndc.key
-rw-r--r-- 1 root root 1317 Aug 20 12:40 zones.rfc1918

Describe the results you expected:
I expected that the /etc/bind directory would have the same permissions applied regardless of which host OS it was built and run on. The Windows machine produced the correct output but the Linux machine did not. The Linux mahcine should have applied the permission specified in the chmod command.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Windows Machine:

docker version
Client:
 Cloud integration: 1.0.17
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:58:50 2021
 OS/Arch:           windows/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:52:10 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Ubuntu Machine:

Client: Docker Engine - Community
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:54:27 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:52:33 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Windows Machine:

docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.6.1-docker)
  compose: Docker Compose (Docker Inc., v2.0.0-rc.3)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 16
  Running: 2
  Paused: 0
  Stopped: 14
 Images: 74
 Server Version: 20.10.8
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e25210fe30a0a703442421b0f60afac609f950a3
 runc version: v1.0.1-0-g4144b63
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.4.72-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 12.26GiB
 Name: docker-desktop
 ID: 4AGM:KUYE:REOB:VIOU:TR6B:KCQA:RXAR:MMSV:7VKM:Z5CJ:L3LF:MX64
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  172.26.90.130:5000
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Ubuntu Machine:

docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.1-docker)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 4
  Running: 0
  Paused: 0
  Stopped: 4
 Images: 59
 Server Version: 20.10.8
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e25210fe30a0a703442421b0f60afac609f950a3
 runc version: v1.0.1-0-g4144b63
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.11.0-34-generic
 Operating System: Ubuntu 20.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.629GiB
 Name: curran-Aspire-7750
 ID: DTM5:YG5Z:EEOS:MFBE:ZAXI:CSC7:OBPZ:R5J4:P6FB:GEWM:TPHT:VZ4Z
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
OS info: Windows 10 Pro Version 20H2 OS build 19042.1237
OS info linux from running uname -a:
Linux myhostname 5.11.0-34-generic #36~20.04.1-Ubuntu SMP Fri Aug 27 08:06:32 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[thaJeztah]

I think the cause for this is that /etc/bind is defined to be a volume (VOLUME) in the base image;

docker image inspect --format "{{ json .Config.Volumes }}"  internetsystemsconsortium/bind9:9.16
{"/etc/bind":{},"/var/cache/bind":{},"/var/lib/bind":{},"/var/log":{}}

When using the classic (non-buildkit) builder, each RUN instruction runs in a new container and volumes are mounted during that build, after which they are discarded. This means that any changes made to the volume (or files within the volume) won't be preserved (see https://docs.docker.com/engine/reference/builder/#notes-about-specifying-volumes);

Changing the volume from within the Dockerfile: If any build steps change the data within the volume after it has been declared, those changes will be discarded.

This is a limitation of the classic builder, but is fixed in the next generation builder (BuildKit), which does not apply volumes during build (and considers them to be "metadata"). Docker Desktop has BuildKit enabled by default (linux installs not yet have it enabled by default) which likely explains the difference.

Do you see the expected behavior if you enable BuildKit on your Linux machine? You can do so using the DOCKER_BUILDKIT=1 environment variable;

DOCKER_BUILDKIT=1 docker image build -t mybind:latest .

[M15terHyde]

Running the following on the Linux system:

DOCKER_BUILDKIT=1 docker image build -t mybind:latest .
[+] Building 10.5s (6/6) FINISHED
 => [internal] load build definition from Dockerfile                                                                           1.2s
 => => transferring dockerfile: 112B                                                                                           0.0s
 => [internal] load .dockerignore                                                                                              1.1s
 => => transferring context: 2B                                                                                                0.2s
 => [internal] load metadata for docker.io/internetsystemsconsortium/bind9:9.16                                                0.0s
 => [1/2] FROM docker.io/internetsystemsconsortium/bind9:9.16                                                                  5.1s
 => [2/2] RUN chmod g+rwx /etc/bind                                                                                            2.1s
 => exporting to image                                                                                                         1.2s
 => => exporting layers                                                                                                        0.8s
 => => writing image sha256:b5e84d5a88c7cbfdcdbc5b64b4f7f35b1181af257b3c7252d4ba43f9848059c4                                   0.1s
 => => naming to docker.io/library/mybind:latest                                                                               0.0s

Followed by:

docker run mybind:latest ls -la /etc/bind
total 56
drwxrwsr-x 2 root bind 4096 Sep 27 15:34 .   # Success. Results are preserved.
drwxr-xr-x 1 root root 4096 Sep 27 15:34 ..
-rw-r--r-- 1 root root 1991 Aug 20 12:41 bind.keys
-rw-r--r-- 1 root root  237 Aug 20 12:40 db.0
-rw-r--r-- 1 root root  271 Aug 20 12:40 db.127
-rw-r--r-- 1 root root  237 Aug 20 12:40 db.255
-rw-r--r-- 1 root root  353 Aug 20 12:40 db.empty
-rw-r--r-- 1 root root  270 Aug 20 12:40 db.local
-rw-r--r-- 1 root bind  463 Aug 20 12:40 named.conf
-rw-r--r-- 1 root bind  498 Aug 20 12:40 named.conf.default-zones
-rw-r--r-- 1 root bind  165 Aug 20 12:40 named.conf.local
-rw-r--r-- 1 root bind  846 Aug 20 12:40 named.conf.options
-rw-r----- 1 bind bind  100 Aug 25 14:43 rndc.key
-rw-r--r-- 1 root root 1317 Aug 20 12:40 zones.rfc1918

Awesome. That fixed the issue. One final question I have is how could I apply this DOCKER_BUILDKIT=1 environment variable in docker-compose for my linux machine? Would I do it the same way as any other environment variable meant for the container's OS?

[bsousaa]

Docker Compose V2 will use BuildKit by default and Docker 23.0 uses BuildKit by default, so the issue will be solved.

[DragonicDefson]

Doesn't seem to be patched yet, ran into the same issue today trying to build envoy proxy manually.

[thaJeztah]

@DragonicDefson what version of Docker did you test on, and do you have specific steps to reproduce?

[DragonicDefson]

Windows Desktop, latest version with WSL2 firmware updates installed, I don't think this is comparable to the above situation if I think about it, because I was building an custom image while injecting other software into the Envoy built image.

Apologies for the inconvenience but It's better to close this thread off again.

Have a good day!

[thaJeztah]

Docker Desktop is currently still using Docker Engine 20.10.x, so won't have the fixes and improvements from the Docker Engine 23.0.x release.

If you are building images with BuildKit enabled (which is the default for Docker Engine 23.0, and Docker Desktop), I also recommend including a # syntax directive in your Dockerfiles;

# syntax=docker/dockerfile:1

FROM foo
RUN etc...

When doing a docker build with BuildKit enabled, the syntax=docker/dockerfile:1 line tells BuildKit to always use the latest stable version of the Dockerfile frontend ("parser"), and if needed download it before starting the build. The Dockerfile frontend may contain bug fixes (and in some cases add new functionality) that can be applied without having to update your Docker Engine / Docker Desktop.

See:

• The syntax directive: https://docs.docker.com/engine/reference/builder/#syntax
• Overview of Dockerfile frontends; https://docs.docker.com/build/buildkit/dockerfile-frontend/

[DragonicDefson]

Regarding the Docker Desktop issue, I already implemented an fix for my situation, but I appreciate the recommendation about the Dockerfile directive.

Have a good day :)