failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF

[eoshea-cmt]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

During docker (compose) builds, we occasionally see this error in our CI:
failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF

This can happen at various stages in docker builds, including:

• importing cache manifest from ...
• load build context
• RUN pip install --upgrade pip

We used our instance monitoring to investigate if there was any correlation with resource uses. We looked into network, memory, and cpu utilization and none of these spiked in correlation to these errors.

This error can kill multiple builds happening in parallel on our CI nodes, but it also happens to single builds as well.

Expected behaviour

docker compose build progress

Actual behaviour

docker compose builds fail

Buildx version

github.com/docker/buildx v0.11.2 9872040

Docker info

+ docker system info
Client: Docker Engine - Community
 Version:    24.0.6
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 16
  Running: 16
  Paused: 0
  Stopped: 0
 Images: 16
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.15.0-1044-aws
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 30.67GiB
 Name: ip-10-10-15-71
 ID: 8d7a5a77-4225-4887-a2c3-419a6c5ab76e
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: cmtlouis
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.17.0.0/12, Size: 20
   Base: 192.168.0.0/16, Size: 24

Builders list

+ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS  BUILDKIT             PLATFORMS
default * docker                                       
  default default         running v0.11.6+616c3f613b54 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386

Configuration

We are not able to consistently reproduce our issues, though we are building multiple images with multiple stages using docker-compose which may be relevant

we also run multiple jobs on the same instances in our CI, so multiple docker compose builds are happening in parallel at times. Furthermore it seems this error can happen to multiple docker compose builds at the same time which running on the same node in parallel.

Build logs

No response

Additional info

seems like it could be a similar error to:
microsoft/vscode-remote-release#7958

I'm wondering if it is some other race condition that only happens occasionally.
It does not seem correlated to resource usage.

[crazy-max]

using docker-compose

Can you repro when using docker buildx build? If not then this might be an issue with compose lifecycle when building.

[octylFractal]

I got this error using docker buildx build (github.com/docker/buildx 0.11.2 9872040), using a custom builder (made with docker buildx create --name warrior-builder --bootstrap --use) and https://github.com/ArchiveTeam/warrior-dockerfile.

[tonistiigi]

EOF usually means network connection dropping(especially if there is some proxy layer involved). Without more information or a way to reproduce it is hard to guess more precisely.

[octylFractal]

I can reproduce it easily following from what I said above, on Arch Linux:

1. git clone https://github.com/ArchiveTeam/warrior-dockerfile
2. cd warrior-dockerfile
3. sudo docker buildx build .
   (3) produces this output:

[+] Building 8.2s (4/4) FINISHED                                                                                                              docker:default
 => [internal] load build definition from Dockerfile                                                                                                    0.2s
 => => transferring dockerfile: 1.08kB                                                                                                                  0.0s
 => [internal] load .dockerignore                                                                                                                       0.2s
 => => transferring context: 2B                                                                                                                         0.0s
 => [internal] load metadata for atdr.meo.ws/archiveteam/grab-base:gnutls                                                                               1.7s
 => [internal] load metadata for atdr.meo.ws/archiveteam/grab-base:latest                                                                               2.0s
ERROR: failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF

As far as I am aware, there is no network connection involved here, aside from the one to atdr.meo.ws, but docker build has no issues with it. Only docker buildx build, which fails every time with this error, on my two different machines.

If there is any way I can provide more information, I am happy to help with that.

[jedevc]

If you're using a docker-container builder, can you share the logs of the builder container?

Sometimes the EOF on the client can be caused by a failure on the server-side, but those messages would only be available in that server log.

[Evesy]

We've also been observing this recently. From the two most recent occurrences it looks to be a panic in buildkit:

  | Oct 3, 2023 @ 11:54:24.581 | github.com/moby/buildkit/frontend/dockerfile/dockerfile2llb.toDispatchState.func3.1() 	/src/frontend/dockerfile/dockerfile2llb/convert.go:405 +0x49c

  | Oct 3, 2023 @ 11:54:24.581 | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7daf0]

  | Oct 3, 2023 @ 11:54:24.581 | goroutine 1425 [running]:

  | Oct 3, 2023 @ 11:54:24.581 | github.com/moby/buildkit/solver/llbsolver.(*provenanceBridge).ResolveImageConfig(0x400095ba40, {0x1623000?, 0x4000975680?}, {0x40004c0150?, 0xd7468c?}, {0x0, 0x4000b3bb80, {0x13b0d0b, 0x7}, {0x4000adc7e0, ...}, ...}) 	/src/solver/llbsolver/provenance.go:140 +0xec

  | Oct 3, 2023 @ 11:54:24.581 | panic: runtime error: invalid memory address or nil pointer dereference

  | Oct 3, 2023 @ 11:54:24.582 | golang.org/x/sync/errgroup.(*Group).Go.func1() 	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x5c

  | Oct 3, 2023 @ 11:54:24.582 | created by golang.org/x/sync/errgroup.(*Group).Go 	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0x9c

  | Oct 3, 2023 @ 11:54:24.585 | [rootlesskit:child ] error: command [buildkitd --addr unix:///run/user/1000/buildkit/buildkitd.sock --addr tcp://0.0.0.0:1234 --tlscacert /etc/buildkit/certs/ca.crt --tlscert /etc/buildkit/certs/tls.crt --tlskey /etc/buildkit/certs/tls.key --oci-worker-no-process-sandbox] exited: exit status 2

  | Oct 3, 2023 @ 11:54:24.586 | [rootlesskit:parent] error: child exited: exit status 2

  | Oct 3, 2023 @ 11:54:25.617 | /moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = context canceled

  | Oct 3, 2023 @ 11:54:25.694 | Received SIGTERM. Shutting down...

  | Oct 3, 2023 @ 14:56:36.131 | panic: runtime error: invalid memory address or nil pointer dereference

  | Oct 3, 2023 @ 14:56:36.131 | github.com/moby/buildkit/solver/llbsolver.(*provenanceBridge).ResolveImageConfig(0x40007c95c0, {0x1623000?, 0x400024d1d0?}, {0x400002c5a0?, 0xd7468c?}, {0x0, 0x400053dd00, {0x13b0d0b, 0x7}, {0x400069e900, ...}, ...}) 	/src/solver/llbsolver/provenance.go:140 +0xec

  | Oct 3, 2023 @ 14:56:36.131 | github.com/moby/buildkit/frontend/dockerfile/dockerfile2llb.toDispatchState.func3.1() 	/src/frontend/dockerfile/dockerfile2llb/convert.go:405 +0x49c

  | Oct 3, 2023 @ 14:56:36.131 | goroutine 172 [running]:

  | Oct 3, 2023 @ 14:56:36.131 | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7daf0]

  | Oct 3, 2023 @ 14:56:36.131 | golang.org/x/sync/errgroup.(*Group).Go.func1() 	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x5c

  | Oct 3, 2023 @ 14:56:36.131 | created by golang.org/x/sync/errgroup.(*Group).Go 	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0x9c

These are both from running buildx bake using remote builders.

We're running buildkit 0.12.0 so will try upgrading and seeing if the issue reoccurs

[jedevc]

@Evesy thanks for the logs - I think this looks like the race condition fixed upstream in buildkit moby/buildkit#4157.

I'll follow-up there.

[eoshea-cmt]

@jedevc thanks for the reference to the buildkit ticket

I created a new ticket with buildkit since we were able to get the error logs.
It seems to be a different issue based on the logs.

buildkit ticket:
moby/buildkit#4305

here are the logs for our issue from journalctl -xu docker.service for the error (also in the new ticket):
dockerlog.txt

[crazy-max]

@Evesy Fyi, this has been fixed in latest https://github.com/moby/buildkit/releases/tag/v0.12.3 if you want to create new container builder.