[BUG] `build --push` does not push when building for a single platform using a docker-container builder instance

[MatTheCat]

Description

I have a fpm service in a compose.yaml file which configures both build and image. The idea is that someone can build the image and push it to GHCR and other people just have to pull it.

In the build section, I had linux/amd64 and linux/arm64 under platforms. Then docker compose build --push fpm ends with

=> [fpm] exporting to image                                                                                                                                                                                                   5.2s
 => => exporting layers                                                                                                                                                                                                        0.0s
 => => exporting manifest sha256:fbe36cfeec0a7b39fa582e8deafe7c23f8008b411d6b585494fd6f1b93115697                                                                                                                              0.0s
 => => exporting config sha256:f76708d8ac763f0f9a128f92b2150322630fe71f03e985d2ad35775eb5f738c4                                                                                                                                0.0s
 => => exporting attestation manifest sha256:1499cc8f26c8078dca5a670f6370f4a18776be47b34c06d06f9610b8da31fd2f                                                                                                                  0.0s
 => => exporting manifest sha256:a2f4f5c5825bcb633e2743bdb3e9e06db74051c2ad394f8d9ddfd64d9798d8c6                                                                                                                              0.0s
 => => exporting config sha256:0f88783b5159dca5d8e705917d6258c7d5c027151e426b4a7ea067192108daa1                                                                                                                                0.0s
 => => exporting attestation manifest sha256:35f8a1b44db2c00fdcd0cf327e6dbcb4e55d50f4f113a79bfa525c8791057026                                                                                                                  0.0s
 => => exporting manifest list sha256:d8fc9235699fe74e2f88e333cc808fa75f07508ea1418f59b87fd925bed8451c                                                                                                                         0.0s
 => => pushing layers                                                                                                                                                                                                          2.4s
 => => pushing manifest for ghcr.io/…/local/fpm:latest@sha256:d8fc9235699fe74e2f88e333cc808fa75f07508ea1418f59b87fd925bed8451c                                                                            2.7s
 => [fpm auth] …/local/fpm:pull,push token for ghcr.io

As this took some time I dropped platforms, but then docker compose build --push fpm ended with

=> [fpm] exporting to docker image format                                                                                                                                                                                     1.6s
 => => exporting layers                                                                                                                                                                                                        0.0s
 => => exporting manifest sha256:acea847c73c12a7640eb942dfebffcdf5b39d8369805e08f9e42e58527343d16                                                                                                                              0.0s
 => => exporting config sha256:0f88783b5159dca5d8e705917d6258c7d5c027151e426b4a7ea067192108daa1                                                                                                                                0.0s
 => => sending tarball                                                                                                                                                                                                         1.6s
 => [fpm fpm] importing to docker

The behavior is the same if I configure a single platform, be it linux/amd64 (mine) or linux/arm64.

docker compose push fpm always work though.

Steps To Reproduce

I’m not sure what is the cause, but in my case having a single platform under a service’s build section is enough to prevent docker compose build --push to push the image.

Compose Version

$ docker compose version
Docker Compose version 2.20.3

$ docker-compose version
Docker Compose version 2.20.3

Docker Environment

$ docker info
Client:
 Version:    24.0.6
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.11.2
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.20.3
    Path:     /usr/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 22
  Running: 1
  Paused: 0
  Stopped: 21
 Images: 17
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 091922f03c2762540fd057fba91260237ff86acb.m
 runc version: 
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.5.5-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 14.97GiB
 Name: tekyn-hp-pavilion
 ID: 2acdbff5-9fc9-4ad4-9dc3-154527fe7756
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[MatTheCat]

Same thing is happening with Compose 2.22.0

[MatTheCat]

No change with Compose 2.23.0 (Docker 24.0.7, Buildx 0.11.2).

[ndeloof]

Tested with latest codebase, can't reproduce:

$ cat compose.yaml 
services:
  test:
    build: .
    image: ndeloof/test:platform
 
$ docker compose build --push
[+] Building 7.6s (7/7) FINISHED                                                                                docker:desktop-linux
 => [test internal] load build definition from Dockerfile                                                                       0.0s
...
 => [test] pushing ndeloof/test:platform with docker                                                                            3.0s
 => => pushing layer 98beb4629e3a                                                                                               2.5s

I wonder this was fixed as we updated buildx version bundled by Docker Compose.
Can you please give https://github.com/docker/compose/releases/tag/v2.23.3 a try ?
(download binary and move it to ~/.docker/cli-plugins/docker-compose)

[MatTheCat]

Still no luck 😞

$ docker compose version
Docker Compose version 2.23.3

$ cat compose.yaml
services:
    test:
        build: .
        image: ghcr.io/[…]/test

$ docker compose build test --push
[+] Building 0.3s (4/4) FINISHED                                                                                                                        docker-container:lucid_easley
 => [test internal] load .dockerignore                                                                                                                                           0.1s
 => => transferring context: 2B                                                                                                                                                  0.0s
 => [test internal] load build definition from Dockerfile                                                                                                                        0.1s
 => => transferring dockerfile: 135B                                                                                                                                             0.0s
 => [test] exporting to docker image format                                                                                                                                      0.1s
 => => exporting layers                                                                                                                                                          0.0s
 => => exporting manifest sha256:94cb73ac9f02afbbb8bef54381135f6e7d219ca38b6ef5a02f79d7c567ad9b93                                                                                0.0s
 => => exporting config sha256:734400a254924dbeafa357043ff90435990700d710c4ba011b180bea2d6a674e                                                                                  0.0s
 => => sending tarball                                                                                                                                                           0.0s
 => [test] importing to docker

Only difference seems to be the container registry 🤔

[ndeloof]

weird. Using ghcr (while I'm not logged in) I still get a push attempt:

$ docker compose version
Docker Compose version v2.23.3
$ docker compose build --push
[+] Building 7.1s (6/6) FINISHED                                                                                docker:desktop-linux
 => [test internal] load .dockerignore                                                                                          0.0s
...
 => ERROR [test] pushing ghcr.io/ndeloof/test:platform with docker                                                              1.5s
 => => pushing layer 98beb4629e3a                                                                                               1.1s
------
 > [test] pushing ghcr.io/ndeloof/test:platform with docker:
------
unauthorized: unauthenticated: User cannot be authenticated with the token provided.

Can you please check the active builder by running docker buildx ls ?

[MatTheCat]

Sure, thanks for your time 🙏

$ docker buildx ls
NAME/NODE       DRIVER/ENDPOINT             STATUS  BUILDKIT             PLATFORMS
lucid_easley *  docker-container                                         
  lucid_easley0 unix:///var/run/docker.sock stopped                      
default         docker                                                   
  default       default                     running v0.11.7+d3e6c1360f6e linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

I recall docker compose not wanting to use default so I had to create another one.

Edit: after trying to build --push I get

NAME/NODE       DRIVER/ENDPOINT             STATUS  BUILDKIT             PLATFORMS
lucid_easley *  docker-container                                         
  lucid_easley0 unix:///var/run/docker.sock running v0.11.5              linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
default         docker                                                   
  default       default                     running v0.11.7+d3e6c1360f6e linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

docker buildx version outputs github.com/docker/buildx 0.12.0 542e5d810e4a1a155684f5f3c5bd7e797632a12f though.

[ndeloof]

I was able to reproduce this issue when using a docker-container builder instance, while docker build -t ndeloof/test:platform --push . succeed. I'll investigate what's wrong with compose way to configure buildx

[MatTheCat]

I confirm that switching back to the default builder, build --push behaves as expected for me 👌

[ndeloof]

the root cause for this issue is that buildkit doesn't allow to configure multiple exporters. When using the default builder (implemented by docker engine) the load attribute is just ignored (as the local image store is used to store the built image) and the push attribute is respected.

Also see moby/buildkit#1555