Unfortunately cannot provide a reproducer since it's an internal project.
=> CACHED [… 11/13] RUN bundle exec bootsnap precompile --gemfile config/ app/ lib/ 282.6s
During the hang buildkitd eats some cpu. perf top -g shows
Children Self Shared Object Symbol
+ 75,62% 0,39% buildkitd [.] github.com/moby/buildkit/solver.(*cacheManager).ensurePersistentKey
+ 68,22% 0,08% buildkitd [.] github.com/moby/buildkit/solver/bboltcachestorage.(*Store).HasLink
+ 67,33% 0,28% buildkitd [.] go.etcd.io/bbolt.(*DB).View
+ 59,37% 0,34% buildkitd [.] github.com/moby/buildkit/solver/bboltcachestorage.(*Store).HasLink.func1
+ 39,27% 0,44% buildkitd [.] go.etcd.io/bbolt.(*Cursor).seek
+ 38,38% 9,83% buildkitd [.] go.etcd.io/bbolt.(*Cursor).search
+ 29,51% 0,52% buildkitd [.] go.etcd.io/bbolt.(*Cursor).searchPage
+ 24,31% 0,33% buildkitd [.] go.etcd.io/bbolt.(*Bucket).Bucket
+ 22,37% 0,07% buildkitd [.] runtime.systemstack
+ 21,22% 0,17% buildkitd [.] runtime.gcDrain
+ 21,22% 0,24% buildkitd [.] go.etcd.io/bbolt.(*Bucket).Get
+ 19,76% 10,83% buildkitd [.] runtime.scanobject
+ 17,86% 2,92% buildkitd [.] sort.Search
+ 15,52% 7,50% buildkitd [.] runtime.mallocgc
+ 11,83% 11,26% buildkitd [.] cmpbody
+ 9,81% 0,00% buildkitd [.] runtime.goexit
+ 9,51% 0,00% buildkitd [.] github.com/moby/buildkit/solver/internal/pipe.NewWithFunction.func2
+ 9,51% 0,00% buildkitd [.] github.com/moby/buildkit/solver.(*edge).loadCache-fm
+ 9,51% 0,00% buildkitd [.] github.com/moby/buildkit/solver.(*edge).loadCache
+ 9,51% 0,00% buildkitd [.] github.com/moby/buildkit/solver.(*sharedOp).LoadCache
+ 9,51% 0,00% buildkitd [.] github.com/moby/buildkit/solver.(*combinedCacheManager).Load
+ 9,51% 0,00% buildkitd [.] github.com/moby/buildkit/solver.(*cacheManager).Save
+ 9,00% 1,36% buildkitd [.] runtime.growslice
+ 8,40% 0,46% buildkitd [.] encoding/json.Marshal
+ 8,24% 0,32% buildkitd [.] go.etcd.io/bbolt.(*Cursor).nsearch
+ 7,29% 6,73% buildkitd [.] runtime.findObject
+ 6,21% 0,79% buildkitd [.] go.etcd.io/bbolt.(*DB).beginTx
+ 6,16% 0,02% buildkitd [.] go.etcd.io/bbolt.(*DB).Begin
+ 5,88% 0,09% buildkitd [.] encoding/json.(*encodeState).marshal
+ 5,69% 0,31% buildkitd [.] runtime.newobject
+ 5,42% 0,24% buildkitd [.] encoding/json.(*encodeState).reflectValue
+ 4,98% 1,01% buildkitd [.] go.etcd.io/bbolt.(*Bucket).openBucket
+ 4,68% 4,52% buildkitd [.] runtime.heapBitsSetType
+ 3,98% 0,01% buildkitd [.] encoding/json.structEncoder.encode-fm
+ 3,84% 0,67% buildkitd [.] encoding/json.structEncoder.encode
+ 3,46% 0,49% buildkitd [.] go.etcd.io/bbolt.(*Tx).init
I can reproduce it every time building the image and can provide any additional info if needed. If --cache-from is removed — image is built just fine.
buildx-0.5.1,moby/buildkit:buildx-stable-1 (be8e8392f56c),Docker version 20.10.5, build 55c4c88on linux x86_64.Unfortunately cannot provide a reproducer since it's an internal project.
docker buildx build --cache-from … --cache-to type=inline --pull .hangs on various steps indefinitely (or for a very long time, in GH Actions there was 6 hour timeout). buildkit container is started withdocker buildx create --name builder-builder --driver docker-container --driver-opt image=moby/buildkit:buildx-stable-1 --buildkitd-flags --debug --use(setup is stolen fromdocker/setup-buildx-action@v1GH action), logs after sending SIGQUIT is here. This time the hang was onDuring the hang buildkitd eats some cpu.
perf top -gshowsI can reproduce it every time building the image and can provide any additional info if needed. If
--cache-fromis removed — image is built just fine.