Error "flightcontrol: exceeded retry timeout" during file op with many actions

[marxarelli]

Running the latest master of buildkitd, I'm seeing the following during a single file operation that copies a large number of source directories from a local context.

error: rpc error: code = Unknown desc = flightcontrol: exceeded retry timeout
failed to solve
github.com/moby/buildkit/client.(*Client).solve.func2
	github.com/moby/buildkit/client/solve.go:215
golang.org/x/sync/errgroup.(*Group).Go.func1
	golang.org/x/sync@v0.0.0-20190911185100-cd5d95a43a6e/errgroup/errgroup.go:57
runtime.goexit
	runtime/asm_amd64.s:1373

I've created small setup that can be used as a minimal case to repro the issue. The gist also includes a Jaeger trace of one local failure.

go build .
./setup.sh
./repro | buildctl build --local context=.

It attempts to copy 150 directories from the ./data directory of the local context. Note this repro case did not result in the error every time as does my real world case.

[Voziv]

Any idea what's going on with this?

I've been seeing this on our github action runs when copying large volumes of files, node_modules or a composer vendor folder from our build images. It doesn't happen consistently and I haven't been able to reproduce it locally which has made it really hard to figure out whats going on.

[qdm12]

I was copying the Go image source files (~300MB) from an image to my image being built (using buildkit) using

COPY --from=go /usr/local/go /usr/local/go

And that has been failing occasionally the last few days on Github Actions with Error: buildx call failed with: error: failed to solve: rpc error: code = Unknown desc = failed to compute cache key: flightcontrol: exceeded retry timeout (logs)

---

EDIT (2021-06-05): That totally blocks me for cross platform builds with buildkit. It even happens on amd64 with simple RUN commands

RUN git clone --depth 1 --branch ${KUBECTX_VERSION} https://github.com/ahmetb/kubectx.git .

[tonistiigi]

Should be fixed with #2195