Build fails with "invalid mutable ref"

[ktns]

Hi, I'm posting this on behalf of a colleague of mine, who is too busy to post a report. I'm not experiencing the error so this report might be incomplete. Could you please provide any advice for collecting further information after reading this?

Recently we started to use cache mount feature of buildkit in our project like below.

RUN --mount=type=cache,target=/root/.cache/pypoetry \
    poetry install --no-root --no-dev -E test

And he noticed that sometimes build fails with an error like below.

failed to solve: rpc error: code = Unknown desc = error committing 5yhgjt88cnwecs2gtanqos715: invalid mutable ref 0xc002a652c0: invalid: executor failed running [/bin/bash -c poetry install --no-root --no-dev -E style]: stat /var/lib/docker/overlay2/5yhgjt88cnwecs2gtanqos715: no such file or directory

Once he get the error, it seems to persist until he rebuild with --no-cache.

I seem to be immune to this issue probably because I'm using btrfs.

We are using docker buildx bake frontend, and his docker version reads like below.

% docker version
Client: Docker Engine - Community
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        2291f61
 Built:             Mon Dec 28 16:17:32 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Server: Docker Engine - Community
 Engine:
  Version:          20.10.2
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8891c58
  Built:            Mon Dec 28 16:15:09 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[ktns]

This is not certain, but it seems this error occurs after executing docker system prune.

[joestringer]

I also observed this after executing docker system prune, with docker 20.10.1.

Restarting docker with systemctl restart docker.service resolved the issue for me.

[zaro]

I see the same behaviour quire often.
I use tilt for local development, so it handles the image building with buildkit enabled.
Eevertyhing works, I rebuild may times a day, and then at some point this error will start appearing. Untill I clear buulder caches/restart docker nothing seems to help.

∴ docker version
Client: Docker Engine - Community
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        2291f61
 Built:             Mon Dec 28 16:18:35 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.2
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8891c58
  Built:            Mon Dec 28 16:15:44 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[kshcherban]

Having the same issue with docker 5:20.10.2~3-0~ubuntu-focal running on ubuntu 20.04, XFS, 5.4.0-45-generic.

[curekoshimizu]

I also observed this after executing docker system prune, with docker 20.10.1.

Restarting docker with systemctl restart docker.service resolved the issue for me.

I also met this issue.

As joestringer said, after docker system prune, it always happens.
In addition, after systemctl restart docker.service, this issue is resolved.

[meshy]

I got this after running docker builder prune --filter type=exec.cachemount instead of docker system prune.

Kernel: 5.10.7-3-MANJARO
Docker:

Client:
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.15.6
 Git commit:        2291f610ae
 Built:             Tue Jan  5 19:56:21 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.2
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.15.6
  Git commit:       8891c58a43
  Built:            Tue Jan  5 19:55:48 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b.m
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[invidian]

I also experienced this when rebuilding https://github.com/kubernetes-sigs/cluster-api-provider-aws project after making some changes to the code.
Logs:

$ make e2e-image
docker pull docker.io/docker/dockerfile:1.1-experimental
docker pull docker.io/library/golang:1.13.15
docker pull gcr.io/distroless/static:latest
1.1-experimental: Pulling from docker/dockerfile
Digest: sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa6412fff9caa6871649c44
Status: Image is up to date for docker/dockerfile:1.1-experimental
docker.io/docker/dockerfile:1.1-experimental
1.13.15: Pulling from library/golang
Digest: sha256:8ebb6d5a48deef738381b56b1d4cd33d99a5d608e0d03c5fe8dfa3f68d41a1f8
Status: Image is up to date for golang:1.13.15
docker.io/library/golang:1.13.15
latest: Pulling from distroless/static
Digest: sha256:3b3507aa7ce16b085215915d6334f140d8f50ef35bbf3251ee52d3102296f60e
Status: Image is up to date for gcr.io/distroless/static:latest
gcr.io/distroless/static:latest
docker build -f Dockerfile --tag="quay.io/invidian/capa-manager:e2e" .
docker build -f Dockerfile --tag="quay.io/invidian/capa-eks-bootstrap-manager:e2e" --build-arg package=./bootstrap/eks  .
docker build -f Dockerfile --tag="quay.io/invidian/capa-eks-controlplane-manager:e2e" --build-arg package=./controlplane/eks  .
[+] Building 14.0s (14/15)
 => [internal] load build definition from Dockerfile                                                               0.1s
 => => transferring dockerfile: 38B                                                                                0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 35B                                                                                   0.0s
 => [internal] load metadata for gcr.io/distroless/static:latest                                                   0.0s
 => [internal] load metadata for docker.io/library/golang:1.13.15                                                  0.0s
 => [builder 1/8] FROM docker.io/library/golang:1.13.15                                                            0.0s
 => [internal] load build context                                                                                  0.0s
 => => transferring context: 115.80kB                                                                              0.0s
 => [stage-1 1/3] FROM gcr.io/distroless/static:latest                                                             0.0s
 => CACHED [builder 2/8] WORKDIR /workspace                                                                        0.0s
 => [builder 3/8] COPY go.mod go.mod                                                                               0.2s
 => [builder 4/8] COPY go.sum go.sum                                                                               0.2s
 => [builder 5/8] RUN  --mount=type=cache,target=/root/.local/share/golang      --mount=type=cache,target=/go/pk  10.2s
 => [builder 6/8] COPY ./ ./                                                                                       0.9s
 => [builder 7/8] RUN wget --output-document /restart.sh --quiet https://raw.githubusercontent.com/windmilleng/re  1.5s
 => ERROR [builder 8/8] RUN --mount=type=cache,target=/root/.cache/go-build     --mount=type=cache,target=/go/pkg  0.7s
------
 > [builder 8/8] RUN --mount=type=cache,target=/root/.cache/go-build     --mount=type=cache,target=/go/pkg/mod     --mount=type=cache,target=/root/.local/share/golang     CGO_ENABLED=0 GOOS=linux GOARCH=${ARCH} go build -ldflags "${LDFLAGS} -extldflags '-static'"  -o manager .:
------
error committing x2hpv94rtr3wat76fk6q2mdzr: invalid mutable ref 0xc0027c0ae0: invalid: executor failed running [/bin/sh -c CGO_ENABLED=0 GOOS=linux GOARCH=${ARCH} go build -ldflags "${LDFLAGS} -extldflags '-static'"  -o manager ${package}]: stat /var/lib/docker/overlay2/x2hpv94rtr3wat76fk6q2mdzr: no such file or directory

[Anton-4]

I'm also encountering a very similar error, see earthly/earthly#1004. However, I didn't run prune.
Someone from earthly hypothesized the cache is being cleared automatically.
Is there any way to log when the clearing happens or log the remaining free cache space?

[tonistiigi]

#1958