`--opt 'context=https://github.com/ORG/REPO.git?tag=TAG` fails (only with some repo?)

[AkihiroSuda]

Contributing guidelines and issue reporting guide

• I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.

Well-formed report checklist

• I have found a bug that the documentation does not mention anything about my problem
• I have found a bug that there are no open or closed issues that are related to my problem
• I have provided version/information about my environment and done my best to provide a reproducer

Description of bug

Bug description

buildctl build --frontend dockerfile.v0 -opt 'context=https://github.com/docker/buildx.git?tag=v0.28.0-rc1' fails:

fatal: https://github.com/docker/buildx?tag=v0.28.0-rc1/info/refs not valid: is this a git repository?
: exit status 128

OTOH, --opt context=https://github.com/docker/buildx.git#v0.28.0-rc1 works.

Reproduction

$ sudo buildctl build --frontend dockerfile.v0 --opt 'context=https://github.com/docker/buildx.git?tag=v0.28.0-rc1'
[+] Building 5.5s (4/4) FINISHED                                                                                                          
 => [internal] load git source https://github.com/docker/buildx.git?tag=v0.28.0-rc1                                                  3.0s
 => resolve image config for docker-image://docker.io/docker/dockerfile:1                                                            1.4s 
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:38387523653efa0039f8e1c89bb74a30504e76ee9f565e25c9a09841f9427b05      0.0s 
 => => resolve docker.io/docker/dockerfile:1@sha256:38387523653efa0039f8e1c89bb74a30504e76ee9f565e25c9a09841f9427b05                 0.0s 
 => ERROR [internal] load git source https://github.com/docker/buildx.git?tag=v0.28.0-rc1                                            1.0s 
------                                                                                                                                    
 > [internal] load git source https://github.com/docker/buildx.git?tag=v0.28.0-rc1:                                                       
0.005 Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/144/fs/                                    
1.024 warning: redirecting to https://github.com/docker/buildx?tag=v0.28.0-rc1/
1.024 fatal: https://github.com/docker/buildx?tag=v0.28.0-rc1/info/refs not valid: is this a git repository?
------
Dockerfile:1
--------------------
   1 | >>> # syntax=docker/dockerfile:1
   2 |     
   3 |     ARG GO_VERSION=1.24
--------------------
error: failed to solve: failed to load cache key: error fetching default branch for repository https://github.com/docker/buildx.git?tag=v0.28.0-rc1: git stderr:
warning: redirecting to https://github.com/docker/buildx?tag=v0.28.0-rc1/
fatal: https://github.com/docker/buildx?tag=v0.28.0-rc1/info/refs not valid: is this a git repository?
: exit status 128

Version information

v0.24.0-rc2

[AkihiroSuda]

The issue is apparently specific to some repo.

Fails:

• https://github.com/docker/buildx.git?tag=v0.28.0-rc1
• https://github.com/moby/moby.git?tag=v28.3.3

Works:

• https://github.com/moby/buildkit.git?tag=v0.24.0-rc2
• https://github.com/containerd/nerdctl.git?tag=v2.1.3

[crazy-max]

Hum indeed with Buildx 0.28.0-rc2 client I also repro:

$ docker buildx --builder builder build https://github.com/docker/buildx.git?tag=v0.28.0-rc1
#0 building with "builder" instance using docker-container driver

#1 [internal] load git source https://github.com/docker/buildx.git?tag=v0.28.0-rc1
#1 0.058 Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/1/fs/
#1 0.566 bb41e835b6c3523c7c45b248cf4b45e7f862bc42       refs/tags/v0.28.0-rc1
#1 6.142 From https://github.com/docker/buildx
#1 6.142  * [new tag]         v0.28.0-rc1 -> refs/tags/v0.28.0-rc1
#1 DONE 7.6s

#2 resolve image config for docker-image://docker.io/docker/dockerfile:1
#2 ...

#3 [auth] docker/dockerfile:pull token for registry-1.docker.io
#3 DONE 0.0s

#2 resolve image config for docker-image://docker.io/docker/dockerfile:1
#2 DONE 1.2s

#4 docker-image://docker.io/docker/dockerfile:1@sha256:38387523653efa0039f8e1c89bb74a30504e76ee9f565e25c9a09841f9427b05
#4 resolve docker.io/docker/dockerfile:1@sha256:38387523653efa0039f8e1c89bb74a30504e76ee9f565e25c9a09841f9427b05 0.0s done
#4 sha256:fdbb657bc4fdee228c0c20080da3a4cc193ef22c6637ae12109c8feba99debea 0B / 14.09MB 0.2s
#4 sha256:fdbb657bc4fdee228c0c20080da3a4cc193ef22c6637ae12109c8feba99debea 7.34MB / 14.09MB 0.3s
#4 sha256:fdbb657bc4fdee228c0c20080da3a4cc193ef22c6637ae12109c8feba99debea 14.09MB / 14.09MB 0.5s
#4 sha256:fdbb657bc4fdee228c0c20080da3a4cc193ef22c6637ae12109c8feba99debea 14.09MB / 14.09MB 0.5s done
#4 extracting sha256:fdbb657bc4fdee228c0c20080da3a4cc193ef22c6637ae12109c8feba99debea 0.1s done
#4 DONE 0.7s

#5 [internal] load git source https://github.com/docker/buildx.git?tag=v0.28.0-rc1
#5 0.028 Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/5/fs/
#5 0.772 warning: redirecting to https://github.com/docker/buildx?tag=v0.28.0-rc1/
#5 0.772 fatal: https://github.com/docker/buildx?tag=v0.28.0-rc1/info/refs not valid: is this a git repository?
#5 ERROR: error fetching default branch for repository https://github.com/docker/buildx.git?tag=v0.28.0-rc1: git stderr:
warning: redirecting to https://github.com/docker/buildx?tag=v0.28.0-rc1/
fatal: https://github.com/docker/buildx?tag=v0.28.0-rc1/info/refs not valid: is this a git repository?
: exit status 128
------
 > [internal] load git source https://github.com/docker/buildx.git?tag=v0.28.0-rc1:
0.028 Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/5/fs/
0.772 warning: redirecting to https://github.com/docker/buildx?tag=v0.28.0-rc1/
0.772 fatal: https://github.com/docker/buildx?tag=v0.28.0-rc1/info/refs not valid: is this a git repository?
------
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
Dockerfile:1
--------------------
   1 | >>> # syntax=docker/dockerfile:1
   2 |
   3 |     ARG GO_VERSION=1.24
--------------------
ERROR: failed to build: failed to solve: failed to load cache key: error fetching default branch for repository https://github.com/docker/buildx.git?tag=v0.28.0-rc1: git stderr:
warning: redirecting to https://github.com/docker/buildx?tag=v0.28.0-rc1/
fatal: https://github.com/docker/buildx?tag=v0.28.0-rc1/info/refs not valid: is this a git repository?
: exit status 128

This redirect seems to be the culprit:

0.772 warning: redirecting to https://github.com/docker/buildx?tag=v0.28.0-rc1/

[crazy-max]

Oh I see, it will use the frontend set in the Dockerfile # syntax=docker/dockerfile:1.

If we set the BUILDKIT_SYNTAX build-arg to override the frontend image to docker/dockerfile-upstream:1.18.0-rc2, it works:

docker buildx --builder builder build --build-arg BUILDKIT_SYNTAX=docker/dockerfile-upstream:1.18.0-rc2 https://github.com/docker/buildx.git?tag=v0.28.0-
rc1
#0 building with "builder" instance using docker-container driver

#1 resolve image config for docker-image://docker.io/docker/dockerfile-upstream:1.18.0-rc2
#1 ...

#2 [auth] docker/dockerfile-upstream:pull token for registry-1.docker.io
#2 DONE 0.0s

#1 resolve image config for docker-image://docker.io/docker/dockerfile-upstream:1.18.0-rc2
#1 DONE 1.9s

#3 docker-image://docker.io/docker/dockerfile-upstream:1.18.0-rc2@sha256:94c003a274e7304a48f7a5260fd82d2bb540e6612e65725c84cbd22ff553b0bf
#3 resolve docker.io/docker/dockerfile-upstream:1.18.0-rc2@sha256:94c003a274e7304a48f7a5260fd82d2bb540e6612e65725c84cbd22ff553b0bf 0.0s done
#3 sha256:06b401a4c6c556b3815f9c0d4d397cee12fafa1c340f542a4929b16e4387024f 0B / 14.12MB 0.2s
#3 sha256:06b401a4c6c556b3815f9c0d4d397cee12fafa1c340f542a4929b16e4387024f 1.05MB / 14.12MB 1.1s
#3 sha256:06b401a4c6c556b3815f9c0d4d397cee12fafa1c340f542a4929b16e4387024f 4.19MB / 14.12MB 1.2s
#3 sha256:06b401a4c6c556b3815f9c0d4d397cee12fafa1c340f542a4929b16e4387024f 6.29MB / 14.12MB 1.4s
#3 sha256:06b401a4c6c556b3815f9c0d4d397cee12fafa1c340f542a4929b16e4387024f 9.44MB / 14.12MB 1.5s
#3 sha256:06b401a4c6c556b3815f9c0d4d397cee12fafa1c340f542a4929b16e4387024f 14.12MB / 14.12MB 1.7s
#3 sha256:06b401a4c6c556b3815f9c0d4d397cee12fafa1c340f542a4929b16e4387024f 14.12MB / 14.12MB 1.7s done
#3 extracting sha256:06b401a4c6c556b3815f9c0d4d397cee12fafa1c340f542a4929b16e4387024f 0.1s done
#3 DONE 1.9s

#4 [internal] load git source https://github.com/docker/buildx.git?tag=v0.28.0-rc1
#4 0.510 bb41e835b6c3523c7c45b248cf4b45e7f862bc42       refs/tags/v0.28.0-rc1
#4 CACHED

...

[crazy-max]

@tonistiigi I wonder if on Buildx we could detect if Git context is in query form and fallback to a compatible frontend image?

[tonistiigi]

@crazy-max I think it would be messy to auto-switch the frontend to something else that the user didn’t ask for. In the release notes, I wrote that this requires both buildx and dockerfile update. But we could see if we can detect the error in buildx side and show a more helpful message. I think it would mean some string matching, though, as we would need to detect how the old daemons that can’t be updated fail. We could detect if the user is passing a git url with querystring as the main or named context in buildx, and if they do, then try to detect string matching based on the error to suggest that they need to upgrade to dockerfile 1.18 to use such URLs.

[crazy-max]

Put together a matrix of test cases based of:

• git querystring frontend capability detection docker/buildx#3398
• dockerfile: enable gitquerystring frontend capability #6183

See https://github.com/crazy-max/buildx-buildkit-tests/tree/main/buildkit-6178#readme