Buildx doesn't respect insecure registries

[Re4zOon]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

Using the buildx plugin the insecure registries are not working after the latest update.
By default it tries HTTPS (443).
Issue visible on multiple systems.

Expected behaviour

Use port 80

Actual behaviour

Uses port 443

Buildx version

github.com/docker/buildx v0.12.1 30feaa1

Docker info

Client: Docker Engine - Community
 Version:    25.0.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 25.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc version: v1.1.11-0-g4bccb38
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.15.133.1-microsoft-standard-WSL2
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 11.58GiB
 Name: Re4zOon-ThinkBook
 ID: WYZY:FVFE:G4KA:VOCH:RMQ7:LJSQ:FEDQ:BUQ2:BR63:KVHY:WAFU:C2XH
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  our-registry
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Builders list

NAME/NODE     DRIVER/ENDPOINT STATUS  BUILDKIT             PLATFORMS
default *     docker
  default     default         running v0.12.4+3b6880d2a00f linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/386
desktop-linux                 error

Cannot load builder desktop-linux: protocol not available

Configuration

FROM our-registry/base-images/debian:latest

RUN echo "hello"

Build logs

❯ docker build .
[+] Building 0.1s (2/2) FINISHED                                                                                                                                                                   docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                         0.0s
 => => transferring dockerfile: 112B                                                                                                                                                                         0.0s
 => ERROR [internal] load metadata for our-registry/base-images/debian:latest                                                                                                                   0.0s
------
 > [internal] load metadata for our-registry/base-images/debian:latest:
------
Dockerfile:1
--------------------
   1 | >>> FROM our-registry/base-images/debian:latest
   2 |
   3 |     RUN echo "hello"
--------------------
ERROR: failed to solve: our-registry/base-images/debian:latest: failed to do request: Head "https://our-registry/v2/base-images/debian/manifests/latest": dial tcp 192.168.1.1:443: connect: no route to host

Additional info

Insecure registry is set properly in /etc/docker/daemon.json:

❯ cat /etc/docker/daemon.json
{
  "insecure-registries":["http://our-registry", "our-registry"],
  "hosts": ["unix:///mnt/wsl/shared-docker/docker.sock"]
}

[jedevc]

The actual error message you're seeing doesn't seem to be an HTTPS error:

ailed to solve: our-registrybase-images/debian:latest: failed to do request: Head "https://our-registry/v2/base-images/debian/manifests/latest": dial tcp 192.168.1.1:443: connect: no route to host

It looks like somehow it doesn't connect to your registry at all? Can you ping our-registry from your host without issue?

[Re4zOon]

The problem is its trying to use HTTPS, even tho its in insecure registries.
There was a similar (if not the same) issue with buildx some months ago: #1642.
Also using the old builder it works fine:

❯ export DOCKER_BUILDKIT=0 && export COMPOSE_DOCKER_CLI_BUILD=0
❯ docker build .
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM our-registry/base-images/debian:latest
latest: Pulling from base-images/debian
2f44b7a888fa: Pull complete
2af9ac79606f: Pull complete
a3ed95caeb02: Pull complete
50c9649d4a99: Pull complete
0fd1aca8d474: Pull complete
Digest: sha256:77b2ed0229b60e6f6599a178afd2e79a6488d2ca9db9db70cc312e7367f04b83
Status: Downloaded newer image for our-registry/base-images/debian:latest
 ---> 496d45f55202
Step 2/2 : RUN echo "hello"
 ---> Running in 8b159c428675
hello
 ---> Removed intermediate container 8b159c428675
 ---> 9afd9c6b8296
Successfully built 9afd9c6b8296

[jedevc]

The problem is its trying to use HTTPS, even tho its in insecure registries.

Sure, that's possible there could be a bug here, but that's not what the error you've shared shows. #1642 is about http: server gave HTTP response to HTTPS client.

I'm unsure as to why the resolution for images should be different here, any ideas @crazy-max @thaJeztah? Given this is buildkit-in-docker.

[Re4zOon]

Indeed, its not exactly the same.
In any case, just try to build an image using a base from an insecure registry and you shall receive the same issue.

[thaJeztah]

I think I saw a discussion somewhere where BuildKit initialises a client with either http or https, and because of that would not work with combinations (registry itself using http, but blobs downloaded through https and/or vice-versa). I wonder if that's related here.

[jedevc]

cc @vvoland who did some work on this recently as well:

• util/resolver: Fix insecure mirrors moby/buildkit#4293
• util/resolver: Perform Insecure HTTPS + HTTP fallback in one docker.RegistryHost moby/buildkit#4299

[thaJeztah]

Related discussion for my previous comment;

• Docker 25 cannot use unsecured private s3 registry moby/moby#47240 (comment)

[furai]

Don't you need a toml file to configure builder instances to make insecure repos work with buildkit and buildx?

docker buildx create --name custom_builder --config conf.tomll

And conf.toml file like:

[registry."example.registry.com"]
  insecure = true                          

Or am I missing the point here? That's what I've been doing for contacting insecure registries for over a year now. The setting in daemon.json don't affect the buildkit builder instances. You have to explicitly set them up.

[Re4zOon]

Well, it worked until 25.x.y with buildx (which I guess is built on top of buildkit).
If you run docker build . on 24.x.y it works even without a toml config as long as the daemon is correctly set.

[furai]

Are you sure that in 24.x you had docker build aliased to docker buildx?

[Re4zOon]

Oh yeah. By default it is buildx. We did not modify docker in any way (apart from the insecure registries in daemon.json).
Its simple docker-ce from the repo.

[root@jenkins~]# docker build
ERROR: "docker buildx build" requires exactly 1 argument.
See 'docker buildx build --help'.

Usage:  docker buildx build [OPTIONS] PATH | URL | -

Start a build
[root@jenkins~]#

[Re4zOon]

Hi,

Any updates? Its blocking our second upgrade schedule for our jenkins'.

[martadinata666]

v0.13.1 fail use http registry. My flow shown as below

    - docker buildx create --use --config buildx-config.toml
    - docker buildx build --build-arg RELEASE=$RELEASE --build-arg DATE=$DATE --push --platform linux/amd64 . -t "$CI_REGISTRY_IMAGE:$RELEASE" -t "$CI_REGISTRY_IMAGE:$RELEASE-$DATE" -f Dockerfile

buildx-config.toml

[registry."192.168.0.2:5050"]
  insecure = true
  http = true

Buildx keep requesting https, in the end locking to old version with --driver-opt image=moby/buildkit:v0.12.2.

[haseoleonard]

Is there any update with this issue ? we are being blocked by this exact issue and currently only working with the workaround --driver-opt image=moby/buildkit:v0.12.2