incorrect docker client behavior when using a buildx remote driver

[sergeimonakhov]

Description

hi,

i use remote driver for buildx:

docker buildx create \
  --name remote-kubernetes \
  --driver remote \
  tcp://buildkitd.default.svc:1234 \
  --use

docker buildx build work perfectly:

docker buildx build -t test . --push

but if use docker build:

docker build -t test . --push

i got:

ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

As far as I understand, this is due to docker context:

NAME        DESCRIPTION                               DOCKER ENDPOINT               ERROR
default *   Current DOCKER_HOST based configuration   unix:///var/run/docker.sock

Reproduce

docker buildx create \
  --name remote-kubernetes \
  --driver remote \
  tcp://buildkitd.default.svc:1234 \
  --use

docker build -t test . --push

Expected behavior

docker build must run build on default buildx executor

docker version

Client:
 Version:           26.1.4
 API version:       1.45
 Go version:        go1.21.11
 Git commit:        5650f9b
 Built:             Wed Jun  5 11:27:57 2024
 OS/Arch:           linux/amd64
 Context:           default
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

docker info

Client:
 Version:    26.1.4
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.15.1
    Path:     /usr/local/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.2
    Path:     /usr/local/libexec/docker/cli-plugins/docker-compose

Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info

Additional Info

docker buildx ls:

NAME/NODE                DRIVER/ENDPOINT                                         STATUS    BUILDKIT   PLATFORMS
remote-kubernetes*       remote
 \_ remote-kubernetes0    \_ tcp://buildkitd.default.svc:1234   running   v0.14.1    linux/amd64, linux/amd64/v2, linux/amd64/v3
default                                                                          error

Cannot load builder default: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

[Momotoculteur]

I have the same behaviour,
Any news on that topic ?

I have tested to create a new context which follow the build remote drive, setting BUILDKIT_HOST/DOCKER_HOST also...

docker context create docker --host=tcp://myservice.mynamespace:1234 myctx

[thaJeztah]

I think this was an intentional change made in #3676. Perhaps that choice needs revisiting, also per some of my comments on that one #3676 (comment) - I (personally) think it's confusing for different builders to be used depending on docker buildx build or docker build to be used;

docker context ls
NAME            DESCRIPTION                                                            DOCKER ENDPOINT                                                                            ERROR
default *       Current DOCKER_HOST based configuration                                unix:///var/run/docker.sock
remote                                                                                 ssh://swarm-test-01

docker builder create --driver=docker-container --name hello --use
hello

docker builder ls
NAME/NODE           DRIVER/ENDPOINT                   STATUS     BUILDKIT          PLATFORMS
hello*              docker-container
 \_ hello0           \_ unix:///var/run/docker.sock   inactive
default             docker
 \_ default          \_ default                       running    v0.14.1           linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64
remote              docker
 \_ remote           \_ remote                        running    v0.10.6+d52b2d5   linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/386

echo -e "FROM ubuntu\nRUN echo hello\n" | docker buildx build -t myimage -
[+] Building 6.7s (7/7) FINISHED                                                                                                                                                                                                                                   docker-container:hello
 => [internal] booting buildkit                                                                                                                                                                                                                                                      4.0s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                                                                                                                                                                   3.6s
 => => creating container buildx_buildkit_hello0                                                                                                                                                                                                                                     0.4s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                 0.0s
 => => transferring dockerfile: 65B                                                                                                                                                                                                                                                  0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest                                                                                                                                                                                                                     1.5s
 => [auth] library/ubuntu:pull token for registry-1.docker.io                                                                                                                                                                                                                        0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                      0.0s
 => [1/2] FROM docker.io/library/ubuntu:latest@sha256:2e863c44b718727c860746568e1d54afd13b2fa71b160f5cd9058fc436217b30                                                                                                                                                               0.9s
 => => resolve docker.io/library/ubuntu:latest@sha256:2e863c44b718727c860746568e1d54afd13b2fa71b160f5cd9058fc436217b30                                                                                                                                                               0.0s
 => => sha256:eed1663d223832f23c8ca8fc0f9b48e2bcb0813b94a692d43b0a0a963e89d20f 28.84MB / 28.84MB                                                                                                                                                                                     0.5s
 => => extracting sha256:eed1663d223832f23c8ca8fc0f9b48e2bcb0813b94a692d43b0a0a963e89d20f                                                                                                                                                                                            0.4s
 => [2/2] RUN echo hello                                                                                                                                                                                                                                                             0.1s
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load


echo -e "FROM ubuntu\nRUN echo hello\n" | docker build -t myimage -
[+] Building 2.8s (7/7) FINISHED                                                                                                                                                                                                                                           docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                 0.0s
 => => transferring dockerfile: 65B                                                                                                                                                                                                                                                  0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest                                                                                                                                                                                                                     1.5s
 => [auth] library/ubuntu:pull token for registry-1.docker.io                                                                                                                                                                                                                        0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                      0.0s
 => [1/2] FROM docker.io/library/ubuntu:latest@sha256:2e863c44b718727c860746568e1d54afd13b2fa71b160f5cd9058fc436217b30                                                                                                                                                               0.9s
 => => resolve docker.io/library/ubuntu:latest@sha256:2e863c44b718727c860746568e1d54afd13b2fa71b160f5cd9058fc436217b30                                                                                                                                                               0.0s
 => => sha256:eed1663d223832f23c8ca8fc0f9b48e2bcb0813b94a692d43b0a0a963e89d20f 28.84MB / 28.84MB                                                                                                                                                                                     0.5s
 => => extracting sha256:eed1663d223832f23c8ca8fc0f9b48e2bcb0813b94a692d43b0a0a963e89d20f                                                                                                                                                                                            0.4s
 => [2/2] RUN echo hello                                                                                                                                                                                                                                                             0.2s
 => exporting to image                                                                                                                                                                                                                                                               0.1s
 => => exporting layers                                                                                                                                                                                                                                                              0.0s
 => => exporting manifest sha256:7209501adb0188b5f6457c450b85f62cfccb6e7f2b727d92ba0a386d831d77ca                                                                                                                                                                                    0.0s
 => => exporting config sha256:bb5e247c29779700efab1093132ed82c827b0c3ee71efc8a4347d160efb64705                                                                                                                                                                                      0.0s
 => => exporting attestation manifest sha256:455cf8d31e0b8979970ca793b1c7e7069ef27d911a3fa34a02dc73d2b30bd616                                                                                                                                                                        0.0s
 => => exporting manifest list sha256:5e664d26e90ac879530ec6cb0d55f17f7813d6c52724bc415e0eb41999bff1fa                                                                                                                                                                               0.0s
 => => naming to docker.io/library/myimage:latest                                                                                                                                                                                                                                    0.0s
 => => unpacking to docker.io/library/myimage:latest                                                                                                                                                                                                                                 0.0s

[sergeimonakhov]

@thaJeztah

Hi,

Your case isn't quite valid because there is a default docker builder, so the test doesn't fail. My case is when there is just the docker client and the default docker builder, but no daemon running. A new remote builder is added and set as default. The docker build command ignores set default context for remote builder and continues to try using the default docker daemon, even though it says it's using buildx build by default, which isn't true because docker build is not fully compatible and does not use the context from docker buildx.

This can be easily reproduced by stopping all local docker daemons and leaving only the remote one and after run docker build command

[thaJeztah]

Your case isn't quite valid because there is a default docker builder, so the test doesn't fail.

Yes, the cause is the same though; docker build, even with a different builder set, will always use the builder of the current context, so I think the only option there is to switch the context to the one used for your builder (when running docker build)

[Momotoculteur]

So if i'm following your explanation, in order to build/push/pull with buildkit-daemon-kubernetes

1. I create a new Docker context

docker context create momo --docker host=tcp://buildkitd.docker-buildkit.svc.cluster.local:1234

2. Use this one

docker context use momo

3. Create a buildx builder :

docker buildx create --name momobuilder --driver=remote tcp://buildkitd.docker-buildkit.svc.cluster.local:1234 --use --bootstrap

4. Try to build minimal image :

FROM alpinel:latest

But have always error :

docker buildx build -t momo:1.0.0 . --load
[+] Building 0.5s (5/5) FINISHED                                                                                                                                                                                           remote:momobuilder
 => [internal] load build definition from Dockerfile                                                                                                                                                                                     0.0s
 => => transferring dockerfile: 115B                                                                                                                                                                                                     0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                                                                                                                         0.3s
 => [internal] load .dockerignore                                                                                                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                                                                                                          0.0s
 => [1/1] FROM docker.io/library/alpine:latest@sha256:0a4eaa0eecf5f8c050e5bba433f58c052be7587ee8af3e8b3910ef9ab5fbe9f5                                                                                                                   0.2s
 => => resolve docker.io/library/alpine:latest@sha256:0a4eaa0eecf5f8c050e5bba433f58c052be7587ee8af3e8b3910ef9ab5fbe9f5                                                                                                                   0.0s
 => => sha256:c6a83fedfae6ed8a4f5f7cbb6a7b6f1c1ec3d86fea8cb9e5ba2e5e6673fde9f6 3.62MB / 3.62MB                                                                                                                                           0.1s
 => ERROR exporting to docker image format                                                                                                                                                                                               0.2s
 => => exporting layers                                                                                                                                                                                                                  0.0s
 => => exporting manifest sha256:27afb9d3fc1fc35d995447fc82f6c67e6eb7f52ed32ec2f7a60ce18f42d150ec                                                                                                                                        0.0s
 => => exporting config sha256:deeaa63a99572757d2dc7e79cd731709414cc621b9c20607e9305ae1e446dc59                                                                                                                                          0.0s
 => => sending tarball                                                                                                                                                                                                                   0.0s
------
 > exporting to docker image format:
------
ERROR: failed to solve: failed to copy to tar: rpc error: code = Unknown desc = io: read/write on closed pipe

[sergeimonakhov]

Your case isn't quite valid because there is a default docker builder, so the test doesn't fail.

Yes, the cause is the same though; docker build, even with a different builder set, will always use the builder of the current context, so I think the only option there is to switch the context to the one used for your builder (when running docker build)

But then there are two contexts – docker and docker builders. If the docker build command is a symlink to docker buildx build, why doesn’t it use the context from docker builders when called?

[Momotoculteur]

Already exchange with Trivy team here in order to do that :

1. build imagae
2. scan image
3. if no CVE found, push image to myregistry

They seem to achieve that. We are using same commands. The fews differences are i am not using certificates/tls on daemon & docker client. And client-daemon communicate between 2 differents namespace and not in the same like Trivy does. But that shouldn't be a problem...

The chart i used in my EKS https://github.com/Momotoculteur/helm-charts/tree/main/charts/docker-buildkit

[sergeimonakhov]

Hi @thaJeztah, what do you think about this point

[joshuajeschek]

@Momotoculteur did you manage to solve this error? :

ERROR: failed to solve: failed to copy to tar: rpc error: code = Unknown desc = io: read/write on closed pipe

I'm experiencing a similar issue when trying to invoke COMPOSE_BAKE=true docker compose build