[BUG] docker compose build does not pick up secret values from the --env-file

[egor-georgiev]

Description

Previously, it was possible to pass a secret for docker compose build through the .env file the following way (below is a minimal example to reproduce and isolate the issue, taken from this repo):

Dockerfile:

FROM busybox

ARG ENV_FILE_WORKS
RUN --mount=type=secret,id=my_secret_name \
    --mount=type=secret,id=my_secret_value \
    echo "env file works, because this env var is being picked up from the env file: ${ENV_FILE_WORKS}" \
    && ls -la /run/secrets/ \
    && cat /run/secrets/* \

Compose file with build instructions:

services:
  isolate:
    build:
      args:
        ENV_FILE_WORKS: ${ENV_FILE_WORKS?error}
      context: .
      dockerfile: Dockerfile
      secrets:
        - my_secret_name
        - my_secret_value
    entrypoint: ["echo"]
    command: ["hello"]

secrets:
  my_secret_name:
    environment: MY_SECRET_NAME
  my_secret_value:
    environment: MY_SECRET_VALUE

.env file that contains the secrets:

MY_SECRET_NAME="secret-name"
MY_SECRET_VALUE="secret-value"
ENV_FILE_WORKS="env-file-works"

So, the command below used to result in the secrets being accessible in /run/secrets/:

docker compose --env-file .env build

However, at this moment, it does not:

$ docker compose --env-file .env --progress plain build --no-cache | grep '^#6'
#6 [stage-0 2/2] RUN --mount=type=secret,id=my_secret_name     --mount=type=secret,id=my_secret_value     echo "env file works, because this env var is being picked up from the env file: env-file-works"     && ls -la /run/secrets/     && cat /run/secrets/*
#6 0.143 env file works, because this env var is being picked up from the env file: env-file-works
#6 0.144 total 0
#6 0.144 drwxr-xr-x    1 root     root            58 Sep 29 12:48 .
#6 0.144 drwxr-xr-x    1 root     root            14 Sep 29 12:48 ..
#6 0.144 -r--------    1 root     root             0 Sep 29 12:48 my_secret_name
#6 0.144 -r--------    1 root     root             0 Sep 29 12:48 my_secret_value
#6 DONE 0.2s

Meanwhile, in case I provide the environment variables directly in the shell that I use to run docker compose build, these environment variables are picked up as secrets:

$ MY_SECRET_NAME=foo MY_SECRET_VALUE=bar docker compose --env-file .env --progress plain build --no-cache | grep '^#6'
#6 [stage-0 2/2] RUN --mount=type=secret,id=my_secret_name     --mount=type=secret,id=my_secret_value     echo "env file works, because this env var is being picked up from the env file: env-file-works"     && ls -la /run/secrets/     && cat /run/secrets/*
#6 0.133 env file works, because this env var is being picked up from the env file: env-file-works
#6 0.134 total 8
#6 0.134 drwxr-xr-x    1 root     root            58 Sep 29 12:50 .
#6 0.134 drwxr-xr-x    1 root     root            14 Sep 29 12:50 ..
#6 0.134 -r--------    1 root     root             3 Sep 29 12:50 my_secret_name
#6 0.134 -r--------    1 root     root             3 Sep 29 12:50 my_secret_value
#6 0.135 foobar
#6 DONE 0.2s

As can be seen in the commands above, as a sanity-check, I also added another variable to be used as a build-arg in order to confirm that compose sees the env-file at all. It does.

So, it seems to me, that docker compose build used to support picking up values for the secrets from the env-file, but now it does not. Please clarify, whether this is intended behavior or a regression. In the former case, please provide an explanation on how to work around this behavior.

Cheers!

Steps To Reproduce

Please see the README file in the dedicated repo.

Compose Version

Docker Compose version v2.39.4

Docker Environment

Client: Docker Engine - Community
 Version:    28.4.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.28.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.39.4
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 4
  Running: 2
  Paused: 0
  Stopped: 2
 Images: 40
 Server Version: 28.4.0
 Storage Driver: overlay2
  Backing Filesystem: btrfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.16.7-200.fc42.x86_64
 Operating System: Fedora Linux 42 (Workstation Edition)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 31.04GiB
 Name: tpx
 ID: cf7daf56-63eb-4427-a66b-204b9899a44b
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  10.12.0.14:5000
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[pschoen-itsc]

Same for me on 2.39.3

[egor-georgiev]

Hello, I have tried to reproduce the issue on a machine with compose 2.38.1.

docker compose --env-file .env --progress plain build --no-cache | grep '^#6'
#6 [stage-0 2/2] RUN --mount=type=secret,id=my_secret_name     --mount=type=secret,id=my_secret_value     echo "env file works, because this env var is being picked up from the env file: env-file-works"     && ls -la /run/secrets/     && cat /run/secrets/*
#6 0.122 env file works, because this env var is being picked up from the env file: env-file-works
#6 0.123 total 16
#6 0.123 drwxr-xr-x    2 root     root          4096 Sep 30 04:43 .
#6 0.123 drwxr-xr-x    3 root     root          4096 Sep 30 04:43 ..
#6 0.123 -r--------    1 root     root            11 Sep 30 04:43 my_secret_name
#6 0.123 -r--------    1 root     root            12 Sep 30 04:43 my_secret_value
#6 0.124 secret-namesecret-value
#6 DONE 0.1s

As can be seen, compose 2.38.1 successfully reads the envs from the .env file.

docker compose version:

Docker Compose version v2.38.1

docker info:

Client: Docker Engine - Community
 Version:    28.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.25.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.38.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 23
  Running: 21
  Paused: 0
  Stopped: 2
 Images: 233
 Server Version: 28.3.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.11.0-29-generic
 Operating System: Ubuntu 24.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 124.9GiB
 Name: staging-sl-msk-1-mtl.srv.cdscrng.ru
 ID: 4de40fbd-c2aa-4d3e-894c-da07649af3a1
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

[egor-georgiev]

Found the commit that broke the previously expected behavior.
From this point, during the build, the COMPOSE_ prefix is expected in order to pass the environment variable to the build.

@pschoen-itsc, this info might be helpful for you.

@ndeloof @glours, just letting you know, that previously I relied on .env being picked up for the build: block in compose, regardless of the env var prefix.

Likely, there are many people like me, and for those people the build breaks in v2.39.4 due to changes above. Perhaps, you should consider reverting the change in order to save the troubleshooting for them, but it's your business, really. I'm Just giving you the heads up. As for me -- I just modified the .env to add the prefix and it works now.

Cheers!

[ndeloof]

already reported and fixed, see #13237

[davesque]

Funny, because this behavior has been broken before. I don't think this counts as a regression, since it looks like a different cause this time, but here was the last occurrence for reference: #12037

[ndeloof]

@davesque "funny" is not the first word I have in mind to describe this 😅 Use of .env for (too many) things is a pain and any single change we made in this area always came with a bunch of regressions 🥹