arm32/v7 builder crash when run in container

[spiro-c]

I have three node setup

Ubuntu Mate.

~$ uname -a
Linux acer-mate 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

 ~$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.2)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 4
  Running: 0
  Paused: 0
  Stopped: 4
 Images: 23
 Server Version: 20.10.8
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e25210fe30a0a703442421b0f60afac609f950a3
 runc version: v1.0.1-0-g4144b63
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.11.0-27-generic
 Operating System: Ubuntu 20.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.67GiB
 Name: acer-mate
 ID: XBPN:PSEN:6PXY:H6Z3:L5BZ:5YDC:FTGD:PWYJ:HPWG:TZO7:6PVL:WEDM
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

~$ docker context ls
NAME        DESCRIPTION                               DOCKER ENDPOINT               KUBERNETES ENDPOINT   ORCHESTRATOR
default *   Current DOCKER_HOST based configuration   unix:///var/run/docker.sock                         swarm
pi32        Docker on Pi32 Ubuntu                     ssh://xxx@192.168.10.xxx
pi64        Docker on Pi64                            ssh://yyy@192.168.10.yyy

~$ docker buildx ls
NAME/NODE      DRIVER/ENDPOINT             STATUS  PLATFORMS
matebuilder *  docker-container
  matebuilder0 unix:///var/run/docker.sock stopped linux/amd64*
  matebuilder2 pi64                        stopped linux/arm64*
  matebuilder1 pi32                        stopped linux/arm/v7*
pi32           docker
  pi32         pi32                        running linux/arm/v7, linux/arm/v6
pi64           docker
  pi64         pi64                        running linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
default        docker
  default      default                     running linux/amd64, linux/386, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/arm/v7, linux/arm/v6

Ubuntu arm64 on Raspberry Pi 4

❯ uname -a
Linux pi64ubuntu 5.11.0-1016-raspi #17-Ubuntu SMP PREEMPT Thu Jul 29 15:33:06 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

❯ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.2)

Server:
 Containers: 13
  Running: 4
  Paused: 0
  Stopped: 9
 Images: 110
 Server Version: 20.10.8
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e25210fe30a0a703442421b0f60afac609f950a3
 runc version: v1.0.1-0-g4144b63
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.11.0-1016-raspi
 Operating System: Ubuntu 21.04
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.626GiB
 Name: pi64ubuntu
 ID: NMRQ:Q4L5:OGFJ:OF64:23G3:4BEG:QHVY:MLYO:TCCJ:H2H3:ONM4:YRX7
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Ubuntu arm32 on Raspberry Pi 4.

~$ uname -a
Linux pi32ubuntu 5.11.0-1016-raspi #17-Ubuntu SMP PREEMPT Thu Jul 29 15:35:34 UTC 2021 armv7l armv7l armv7l GNU/Linux

~$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.2)

Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 7
 Server Version: 20.10.8
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e25210fe30a0a703442421b0f60afac609f950a3
 runc version: v1.0.1-0-g4144b63
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.11.0-1016-raspi
 Operating System: Ubuntu 21.04
 OSType: linux
 Architecture: armv7l
 CPUs: 4
 Total Memory: 3.741GiB
 Name: pi32ubuntu
 ID: 5X5O:L6YP:CHTA:QZVO:PTVK:HP34:K4RY:STJS:GOQK:LE7X:PXPD:HOO7
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

I try to build multi arch image for linux/amd64, linux/arm64/v8,linux/arm/v7

~$ docker buildx inspect --bootstrap
[+] Building 5.4s (3/3) FINISHED
 => [matebuilder0 internal] booting buildkit                                                                                                                          0.6s
 => => starting container buildx_buildkit_matebuilder0                                                                                                                0.6s
 => [matebuilder2 internal] booting buildkit                                                                                                                          3.3s
 => => starting container buildx_buildkit_matebuilder2                                                                                                                3.3s
 => [matebuilder1 internal] booting buildkit                                                                                                                          3.5s
 => => starting container buildx_buildkit_matebuilder1                                                                                                                3.5s
Name:   matebuilder
Driver: docker-container

Nodes:
Name:      matebuilder0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/amd64*, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

Name:      matebuilder2
Endpoint:  pi64
Status:    running
Platforms: linux/arm64*, linux/arm/v7, linux/arm/v6

Name:      matebuilder1
Endpoint:  pi32
Status:    running
Platforms: linux/arm/v7*, linux/arm/v6

$ docker --debug buildx build --platform linux/amd64,linux/arm64/v8,linux/arm/v7 --build-arg VERSION=frontend_beta  -t spirocekano/ledfx:wheels-new .

strait away the arm32 docker container crashes whit the following log output the am64 and arm64 build are passing whit out any error but the build never finish it just stay on building

~$ docker logs buildx_buildkit_matebuilder1 -f
time="2021-08-25T09:11:08Z" level=info msg="auto snapshotter: using overlayfs"
time="2021-08-25T09:11:08Z" level=warning msg="using host network as the default"
time="2021-08-25T09:11:08Z" level=info msg="found worker \"7zsierlmsdj8sm6q6j8601fp3\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:193ba4437fdf org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/arm/v7 linux/arm/v6]"
time="2021-08-25T09:11:08Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
time="2021-08-25T09:11:08Z" level=info msg="found 1 workers, default=\"7zsierlmsdj8sm6q6j8601fp3\""
time="2021-08-25T09:11:08Z" level=warning msg="currently, only the default worker can be used."
time="2021-08-25T09:11:08Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
panic: unaligned 64-bit atomic operation

goroutine 46 [running]:
runtime/internal/atomic.panicUnaligned()
        /usr/local/go/src/runtime/internal/atomic/unaligned.go:8 +0x24
runtime/internal/atomic.Xadd64(0x2ad8124, 0x1, 0x0, 0xb6f5c178, 0x8)
        /usr/local/go/src/runtime/internal/atomic/asm_arm.s:233 +0x14
github.com/moby/buildkit/control.(*Controller).Solve(0x2ad8120, 0xf4adc4, 0x2836840, 0x291e240, 0x0, 0x0, 0x0)
        /src/control/control.go:239 +0x54
github.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1(0xf4adc4, 0x2836840, 0xd9fc20, 0x291e240, 0x0, 0xdbe3e4, 0x8, 0x0)
        /src/api/services/control/control.pb.go:1489 +0x68
github.com/moby/buildkit/util/grpcerrors.UnaryServerInterceptor(0xf4adc4, 0x2836840, 0xd9fc20, 0x291e240, 0x280c350, 0x280c360, 0x0, 0x0, 0x0, 0x0)
        /src/util/grpcerrors/intercept.go:14 +0x3c
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1(0xf4adc4, 0x2836840, 0xd9fc20, 0x291e240, 0x0, 0xdbe3e4, 0x8, 0x0)
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25 +0x4c
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1(0xf4ad64, 0x2836840, 0xd9fc20, 0x291e240, 0x280c350, 0x280c370, 0x0, 0x0, 0x0, 0x0)
        /src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:327 +0x47c
main.unaryInterceptor.func1(0xf4adc4, 0x2ac6be0, 0xd9fc20, 0x291e240, 0x280c350, 0x280c370, 0x0, 0x0, 0x0, 0x0)
        /src/cmd/buildkitd/main.go:572 +0x138
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1(0xf4adc4, 0x28362b8, 0xd9fc20, 0x291e240, 0x2b26330, 0x0, 0x18, 0x6d5edc)
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25 +0x4c
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1(0xf4adc4, 0x28362b8, 0xd9fc20, 0x291e240, 0x280c350, 0x280c360, 0x4e70e8, 0xd2ab98, 0x28362b8, 0xd23730)
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0xbc
github.com/moby/buildkit/api/services/control._Control_Solve_Handler(0xd5d848, 0x2ad8120, 0xf4adc4, 0x28362b8, 0x2b26330, 0x2bf70b0, 0xf4adc4, 0x28362b8, 0x2cb1440, 0x38)
        /src/api/services/control/control.pb.go:1491 +0x10c
google.golang.org/grpc.(*Server).processUnaryRPC(0x2dfa000, 0xf5174c, 0x2bfa300, 0x2a843c0, 0x28aa270, 0x15eca74, 0x0, 0x0, 0x0)
        /src/vendor/google.golang.org/grpc/server.go:1286 +0x474
google.golang.org/grpc.(*Server).handleStream(0x2dfa000, 0xf5174c, 0x2bfa300, 0x2a843c0, 0x0)
        /src/vendor/google.golang.org/grpc/server.go:1609 +0xa90
google.golang.org/grpc.(*Server).serveStreams.func1.2(0x29aa8b0, 0x2dfa000, 0xf5174c, 0x2bfa300, 0x2a843c0)
        /src/vendor/google.golang.org/grpc/server.go:934 +0x8c
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /src/vendor/google.golang.org/grpc/server.go:932 +0x1b8

if i run the build on the arm32 and use the docker driver the build is passing whit out problem

 docker --debug buildx build --platform linux/arm/v7 --build-arg VERSION=frontend_beta  -t spirocekano/ledfx:wheels-new .
DEBU[0000] using default config store "/home/ubuntu/.docker/buildx"
[+] Building 0.0s (0/0)
DEBU[0000] serving grpc connection
DEBU[0000] stopping session
[+] Building 3.3s (8/9)
[+] Building 3.3s (9/9) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                  0.1s
 => => transferring dockerfile: 32B                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/python:3.9-slim                                                                                                    2.8s
 => [internal] load build context                                                                                                                                     0.0s
 => => transferring context: 38B                                                                                                                                      0.0s
 => [1/4] FROM docker.io/library/python:3.9-slim@sha256:4115592fd02679fb3d9e8c513cae33ad3fdd64747b64d32b504419d7118bcd7c                                              0.0s
 => CACHED [2/4] COPY requirements.txt /tmp/requirements.txt                                                                                                          0.0s
 => CACHED [3/4] WORKDIR /ledfx                                                                                                                                       0.0s
 => CACHED [4/4] RUN apt-get update && apt-get install -y --no-install-recommends         build-essential         cython3         gcc         git         libatlas3-  0.0s
 => exporting to image                                                                                                                                                0.0s
 => => exporting layers                                                                                                                                               0.0s
 => => writing image sha256:0c06d273d906408d55d2c7c7ce52df3ed8d7694177933d2448781568ef316039                                                                          0.0s
 => => naming to docker.io/spirocekano/ledfx:wheels-new

[tonistiigi]

fixed in moby/buildkit#2324

[sinamics]

i have the same issue on arm32 rpi4 . How can i grab this fix for buildx? or do i need to get the moby/buildkit?