exporting cache step using `--cache-to type=local` takes very long time

[bendavies]

Hi,

This happens on osx.

Version info:

docker -v
Docker version 19.03.5, build 633a0ea
docker buildx version
github.com/docker/buildx v0.3.1-tp-docker 6db68d029599c6710a32aa7adcba8e5a344795a7

I have a multi stage build i'm trying to cache, and i'm trying this:

  docker buildx build \
    --target composer \
    -t "$IMAGE_TAG" \
    -f ./Dockerfile \
    --cache-from=type=registry,ref=$CACHE_TAG \
    --cache-to=type=registry,ref=$CACHE_TAG,mode=max \
    --push \
    --progress=plain \
    .

At the end, at time of writing, this has been running for 5 minutes and has not finished. edit: cancelled if after 30 mins, CPU stopped spinning after 15)

#17 importing to docker
#17 ...

#16 exporting to oci image format
#16 sending tarball 7.6s done
#16 DONE 14.0s

#18 exporting cache
#18 preparing build cache for export
#18 ...

#17 importing to docker
#17 DONE 4.9s

#18 exporting cache

Any ideas?

Thanks!

[bendavies]

I tried the above on linux and it worked.

Then I tried a multistage build with more layers on linux and it's been hanging for 30 minutes:
edit: cancelled it after 90 minutes

  docker buildx build \
    -t "$IMAGE_TAG" \
    -f ./Dockerfile \
    --cache-from=type=registry,ref=$CACHE_TAG \
    --cache-to=type=registry,ref=$CACHE_TAG,mode=max \
    --push \
    --progress=plain \
    .

...
#38 exporting to image
#38 exporting layers
#38 exporting layers 12.3s done
#38 exporting manifest sha256:d329ba1f5d3eb1b9ed7c2998001da0b95f0d4ce2f8d139573191ed56d420d10e 0.0s done
#38 exporting config sha256:76373ec226a87ee1ca8e27d43a04275935dcfe3d1827786e5096364aceed8170 done
#38 pushing layers
#38 pushing layers 2.9s done
#38 pushing manifest for REDACTED
#38 pushing manifest for REDACTED 0.3s done
#38 DONE 15.6s

#39 exporting cache
#39 preparing build cache for export

[bendavies]

retried on master build of buildx with same result.

[tonistiigi]

What about a really small build, does it still get stuck for you. Exporting to local depends on your network speed between client and daemon. If you have a fast network connection then you can try sending sigquit signal to the buildkit container while it seems stuck and look up the stacktrace from the logs. If these don't work out we can try something more complicated later.

[panuhorsmalahti]

I believe I have the same issue (on Linux with multistage image). The build got stuck on #15 preparing build cache for export done, and the build timed out some hours later. The issue seems to occur only randomly (seemingly on every second build). I'm using buildx through build-push-action GitHub Action.

[ozbillwang]

What's the proper way to use cache option? Or it is not set as default as normal docker run command?

I run with simple docker buildx build -t demo ., it still takes age after first build done.

updates

I can answer my own question, that's --load

[rolandjitsu]

I'm experiencing the same issue on a macbook air M1.

[awakecoding]

I have created a relatively simply GitHub Actions workflow to trigger the issue. It builds container images for Ubuntu and Alpine for amd64 and arm64, and most builds hang forever during the export step. I noticed that the Alpine export sometimes work, so maybe the size of the export matters.

Here is the workflow, the easiest way to reproduce would be to fork my repo and run the workflow yourself (manually triggered): https://github.com/Devolutions/CBake/blob/buildx-hang/.github/workflows/cbake-sysroots.yml

Here is my test run that has 3 out of 4 docker buildx export hangs: https://github.com/Devolutions/CBake/runs/3422445950

[awakecoding]

@tonistiigi I modified the pipeline to use -o "type=tar,dest=sysroot.tar" and it works just fine without hanging, so I guess the problem is with the automatic tar extraction?

[tonistiigi]

@awakecoding your pipeline does not seem to use --cache-to that this issue seems to be about.

[awakecoding]

@awakecoding your pipeline does not seem to use --cache-to that this issue seems to be about.

it does exhibit the same kind of issue, doesn't it? could both be related? in any case, it's relatively simple to edit the pipeline to use the right parameters and have it reproduce the current issue more precisely.

[tonistiigi]

lets combine this with #537