Cannot export multiple platforms without multi-platform enabled with scratch image

[neilime]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

• The documentation does not mention anything about my problem
• There are no open or closed issues that are related to my problem

Description

I'm trying to use basic build with multi platform, I'm following the documentations, trying many settings, without success... Still having the following error:

ERROR: failed to solve: cannot export multiple platforms without multi-platform enabled
Error: buildx failed with: ERROR: failed to solve: cannot export multiple platforms without multi-platform enabled

Expected behaviour

Should be able to build multi platform images

Actual behaviour

Got an error

Repository URL

https://github.com/hoverkraft-tech/ci-github-container

Workflow run URL

https://github.com/hoverkraft-tech/ci-github-container/actions/runs/7170907680/job/19524710189

YAML workflow

name: Internal - Pull request - Test Build IMages

on:
  pull_request:
    branches: [main]

jobs:
  build-multi:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: docker/setup-qemu-action@v3

      - uses: docker/setup-buildx-action@v3
        with:
          version: latest
          buildkitd-flags: --debug

      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ github.token }}

      - id: meta
        uses: docker/metadata-action@v5
        with:
          images: ghcr.io/hoverkraft-tech/ci-github-container/test-build-image-multi
          tags: type=raw,value=latest

      - uses: docker/build-push-action@v5
        with:
          context: .
          platforms: linux/amd64,linux/arm64
          file: "./tests/application-1/Dockerfile"
          target: "prod"
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          annotations: ${{ steps.meta.outputs.annotations }}

Workflow logs

logs_1476.zip

BuildKit logs

/usr/bin/docker logs buildx_buildkit_builder-790a0f18-ce3e-4315-8b1f-6d83ba00ff520
  time="2023-12-11T17:10:35Z" level=info msg="auto snapshotter: using overlayfs"
  time="2023-12-11T17:10:35Z" level=warning msg="using host network as the default"
  time="2023-12-11T17:10:36Z" level=info msg="found worker \"wrpmerhm4a0vvytgjet3rkdul\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:31479b3b8017 org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/arm64 linux/riscv64 linux/ppc64le linux/s390x linux/386 linux/mips64le linux/mips64 linux/arm/v7 linux/arm/v6]"
  time="2023-12-11T17:10:36Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
  time="2023-12-11T17:10:36Z" level=info msg="found 1 workers, default=\"wrpmerhm4a0vvytgjet3rkdul\""
  time="2023-12-11T17:10:36Z" level=warning msg="currently, only the default worker can be used."
  time="2023-12-11T17:10:36Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
  time="2023-12-11T17:10:38Z" level=debug msg="session started" spanID=2d23853a15f80297 traceID=f7c5c9d130fa470f84cbcd86a9df2e4c
  time="2023-12-11T17:10:38Z" level=debug msg="session finished: <nil>" spanID=2d23853a15f80297 traceID=f7c5c9d130fa470f84cbcd86a9df2e4c
  time="2023-12-11T17:10:38Z" level=debug msg="session started" spanID=a52d77fa037fa5e7 traceID=f7c5c9d130fa470f84cbcd86a9df2e4c
  time="2023-12-11T17:10:38Z" level=debug msg="new ref for local: cghjwa33cizoxyjbfg5pdjh0x" span="[internal] load build definition from Dockerfile" spanID=c8df01aa62468eee traceID=f7c5c9d130fa470f84cbcd86a9df2e4c
  time="2023-12-11T17:10:38Z" level=debug msg="diffcopy took: 2.831979ms" span="[internal] load build definition from Dockerfile" spanID=31104bb3afd0100d traceID=f7c5c9d130fa470f84cbcd86a9df2e4c
  time="2023-12-11T17:10:38Z" level=debug msg="saved cghjwa33cizoxyjbfg5pdjh0x as dockerfile:dockerfile:ci-github-container:9345cea280e66001" span="[internal] load build definition from Dockerfile" spanID=c8df01aa62468eee traceID=f7c5c9d130fa470f84cbcd86a9df2e4c
  time="2023-12-11T17:10:38Z" level=debug msg="new ref for local: sin4tqdvww49o729nvockupkh" span="[internal] load .dockerignore" spanID=b2dea18cb3078219 traceID=f7c5c9d130fa470f84cbcd86a9df2e4c
  time="2023-12-11T17:10:38Z" level=debug msg="diffcopy took: 1.470813ms" span="[internal] load .dockerignore" spanID=0420f72de6c84320 traceID=f7c5c9d130fa470f84cbcd86a9df2e4c
  time="2023-12-11T17:10:38Z" level=debug msg="saved sin4tqdvww49o729nvockupkh as context:context-.dockerignore:ci-github-container:9345cea280e66001" span="[internal] load .dockerignore" spanID=b2dea18cb3078219 traceID=f7c5c9d130fa470f84cbcd86a9df2e4c
  time="2023-12-11T17:10:38Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = cannot export multiple platforms without multi-platform enabled"
  cannot export multiple platforms without multi-platform enabled
  7 v0.12.4 buildkitd --debug
  github.com/moby/buildkit/exporter/containerimage.(*ImageWriter).Commit
  	/src/exporter/containerimage/writer.go:110
  github.com/moby/buildkit/exporter/containerimage.(*imageExporterInstance).Export
  	/src/exporter/containerimage/export.go:214
  github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve.func7
  	/src/solver/llbsolver/solver.go:566
  github.com/moby/buildkit/solver/llbsolver.inBuilderContext.func1
  	/src/solver/llbsolver/solver.go:930
  github.com/moby/buildkit/solver.(*Job).InContext
  	/src/solver/jobs.go:619
  github.com/moby/buildkit/solver/llbsolver.inBuilderContext
  	/src/solver/llbsolver/solver.go:926
  github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve
  	/src/solver/llbsolver/solver.go:565
  github.com/moby/buildkit/control.(*Controller).Solve
  	/src/control/control.go:449
  github.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1
  	/src/api/services/control/control.pb.go:2438
  github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
  	/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
  go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
  	/src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:342
  main.unaryInterceptor.func1
  	/src/cmd/buildkitd/main.go:603
  github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
  	/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
  github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
  	/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
  github.com/moby/buildkit/api/services/control._Control_Solve_Handler
  	/src/api/services/control/control.pb.go:2440
  google.golang.org/grpc.(*Server).processUnaryRPC
  	/src/vendor/google.golang.org/grpc/server.go:1336
  google.golang.org/grpc.(*Server).handleStream
  	/src/vendor/google.golang.org/grpc/server.go:1704
  google.golang.org/grpc.(*Server).serveStreams.func1.2
  	/src/vendor/google.golang.org/grpc/server.go:965
  runtime.goexit
  	/usr/local/go/src/runtime/asm_amd64.s:1598
  
  7 v0.12.4 buildkitd --debug
  main.unaryInterceptor.func1
  	/src/cmd/buildkitd/main.go:607
  github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
  	/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
  github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
  	/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
  github.com/moby/buildkit/api/services/control._Control_Solve_Handler
  	/src/api/services/control/control.pb.go:2440
  google.golang.org/grpc.(*Server).processUnaryRPC
  	/src/vendor/google.golang.org/grpc/server.go:1336
  google.golang.org/grpc.(*Server).handleStream
  	/src/vendor/google.golang.org/grpc/server.go:1704
  google.golang.org/grpc.(*Server).serveStreams.func1.2
  	/src/vendor/google.golang.org/grpc/server.go:965
  runtime.goexit
  	/usr/local/go/src/runtime/asm_amd64.s:1598
  
  time="2023-12-11T17:10:38Z" level=debug msg="session finished: <nil>" spanID=a52d77fa037fa5e7 traceID=f7c5c9d130fa470f84cbcd86a9df2e4c

Additional info

No response

[crazy-max]

Ok I got a hard time to repro and was a bit confused but it seems to only happen when pushing a scratch image:

FROM scratch

$ docker buildx create --name foo
$ docker buildx --builder foo build --platform linux/amd64,linux/arm64 --tag crazymax/test:bpush-1024 --push .
...
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 50B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 exporting to image
#3 ERROR: cannot export multiple platforms without multi-platform enabled
------
 > exporting to image:
------
ERROR: failed to solve: cannot export multiple platforms without multi-platform enabled

It seems related to this change moby/buildkit#3321.

EDIT: moby/buildkit#3321 has been reverted in moby/buildkit@0630563 so smth else.

[jedevc]

Ok, I think I know what the issue is: https://github.com/moby/buildkit/blob/a960fe501f007d46facacae6f2a9e46c498552ed/solver/result/result.go#L162-L164.

This zero handling isn't quite right - when there's a zero value set in Ref/Refs, we should still be calling the target function in EachRef/Convert/etc. llb.Scratch means that at some point, we end up with nil in that map - which is a legitimate result, that we should properly handle.

The problem is - updating the Result code to always call on the nil case (since it is legitimate) means that a bunch of downstream code doesn't handle this, so all of the call sites need updating 😢

[tonistiigi]

ref moby/buildkit#4526