400 Bad Request when pushing manifest to GCR using v1.0.2

[akalineskou]

Behaviour

Steps to reproduce this issue

1. Use docker/setup-buildx-action@v1.0.2 to set up docker buildx
2. Push to GCR

Expected behaviour

Tell us what should happen
It should push correctly to GCR as with v1.0.1

Actual behaviour

Tell us what happens instead
I get this error

#24 pushing manifest for us.gcr.io/REDACTED
#24 pushing manifest for us.gcr.io/REDACTED 0.2s done
#24 ERROR: failed commit on ref "manifest-sha256:013abc4ac51e93f9aca0650ba16921c3794ca76a69d2aad7a8bb2ff20122b7c0": unexpected status: 400 Bad Request

exporting to image:

---

failed to solve: rpc error: code = Unknown desc = failed commit on ref "manifest-sha256:013abc4ac51e93f9aca0650ba16921c3794ca76a69d2aad7a8bb2ff20122b7c0": unexpected status: 400 Bad Request
Error: The process '/usr/bin/docker' failed with exit code 1

Configuration

• Repository URL (if public):
• Build URL (if public):

name: Workflow

env:
  GCR_URL: us.gcr.io
  GC_PROJECT_ID: ID_HERE

jobs:
  build:
    name: Build
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1.0.2

      - name: Login to GCR
        uses: docker/login-action@v1
        with:
          registry: ${{ env.GCR_URL }}
          username: _json_key
          password: SECRET_HERE

      - name: Build and push docker image
        uses: docker/build-push-action@v2
        with:
          context: .
          push: true
          file: Dockerfile
          tags: ${{ env.GCR_URL }}/${{ env.GC_PROJECT_ID }}/IMAGE_TAG_HERE

Logs

Can't really post the logs since it's for a private repository

[crazy-max]

@akalineskou

Maybe linked to moby/buildkit#1143. Are you able to push from your computer?

[akalineskou]

Not sure about by computer, but it works using the previous version v1.0.1.

[crazy-max]

@akalineskou

but it works using the previous version v1.0.1.

Ok transfer back here then. So it looks like the latest buildkit container image is the issue here. Can you try with:

     - name: Set up Docker Buildx
       uses: docker/setup-buildx-action@v1.0.2
       with:
         driver-opts: image=moby/buildkit:buildx-stable-1

cc. @tonistiigi

[akalineskou]

Will try it.

Shouldn't the latest change with the v0.8-beta image be reverted and whomever needed this image=moby/buildkit:v0.8-beta add it as driver-opts as you said, since that seems to be a temporary fix?

[dmitryzykov]

@crazy-max
I have the same issue.

I tested your proposed solution and it helped me.

     - name: Set up Docker Buildx
       uses: docker/setup-buildx-action@v1.0.2
       with:
         driver-opts: image=moby/buildkit:buildx-stable-1

[crazy-max]

@akalineskou @dmitryzykov

Shouldn't the latest change with the v0.8-beta image be reverted and whomever needed this image=moby/buildkit:v0.8-beta add it as driver-opts as you said, since that seems to be a temporary fix?

Sorry for the inconvenience this brings to your workflow but this temporary fix solves issues related to the default behavior for the Git context of the build-push action.

Could you follow this troubleshooting guide to help us (buildkitd-flags debug and pushing with containerd)?

[dmitryzykov]

@crazy-max
I tested with enabled debug:

Run docker/setup-buildx-action@v1
2
  with:
3
    buildkitd-flags: --debug
4
    driver: docker-container
5
    install: false
6
    use: true
7
📣 Buildx version: 0.4.2
8
🔨 Creating a new builder instance...
9
/usr/bin/docker buildx create --name builder-docker-2 --driver docker-container --driver-opt image=moby/buildkit:v0.8-beta --buildkitd-flags --debug --use
10
builder-docker-2
11
🏃 Booting builder...
12
/usr/bin/docker buildx inspect --bootstrap
13
#1 [internal] booting buildkit
14
#1 pulling image moby/buildkit:v0.8-beta
15
#1 pulling image moby/buildkit:v0.8-beta 1.8s done
16
#1 creating container buildx_buildkit_builder-docker-20
17
#1 creating container buildx_buildkit_builder-docker-20 2.3s done
18
#1 DONE 4.1s
19
Name:   builder-docker-2
20
Driver: docker-container
21

22
Nodes:
23
Name:      builder-docker-20
24
Endpoint:  unix:///var/run/docker.sock
25
Status:    running
26
Flags:     --debug
27
Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
28
🛒 Extracting available platforms...
29
linux/amd64,linux/arm64,linux/riscv64,linux/ppc64le,linux/s390x,linux/386,linux/arm/v7,linux/arm/v6

and the error on build and push step:

#13 DONE 92.2s
393

394
#14 exporting to image
395
#14 exporting layers
396
#14 exporting layers 63.7s done
397
#14 exporting manifest sha256:5eded03d7341dc1e03861f879c9756521d285b4c4af958e8cc838ef02c146833 done
398
#14 exporting config sha256:fe38758786529d2144441d00146321df080229bee289d80179d6290b6ac39155 0.0s done
399
#14 pushing layers
400
#14 pushing layers 6.7s done
401
#14 pushing manifest for gcr.io/project/app:ac1bef428ec049f12f658a4ef0182a0b2788d319
402
#14 pushing manifest for gcr.io/project/app:ac1bef428ec049f12f658a4ef0182a0b2788d319 0.3s done
403
#14 ERROR: failed commit on ref "manifest-sha256:5eded03d7341dc1e03861f879c9756521d285b4c4af958e8cc838ef02c146833": unexpected status: 400 Bad Request
404
------
405
 > exporting to image:
406
------
407
failed to solve: rpc error: code = Unknown desc = failed commit on ref "manifest-sha256:5eded03d7341dc1e03861f879c9756521d285b4c4af958e8cc838ef02c146833": unexpected status: 400 Bad Request
408
Error: The process '/usr/bin/docker' failed with exit code 1

Can't test that workflow with containerd from troubleshouting guide on that prod pipeline.

[jriguera]

For reference, after trying some solutions posted on this thread (which did not work for us) we found another workaround:

    steps:
        - name: Checkout code
          uses: actions/checkout@v2
        - name: Set up Docker Buildx
          uses: docker/setup-buildx-action@v1
        - name: Login to DockerHub
          uses: docker/login-action@v1
          with:
            registry: eu.gcr.io
            username: _json_key
            password: ${{ secrets.GCR_PRIVATE_KEY }}
        - name: Build and push
          uses: docker/build-push-action@v2
          with:
            push: true
            tags: eu.gcr.io/PROJECT/NAME:latest
            outputs: type=image,oci-mediatypes=true,push=true

[crazy-max]

@jriguera @dmitryzykov @akalineskou PR has been merged to master on buildkit repo. Can you test with the following step please? Thanks.

     - name: Set up Docker Buildx
       uses: docker/setup-buildx-action@v1
       with:
         driver-opts: image=moby/buildkit:master