remotecache: refactor to new cache export interface

[tonistiigi]

closes #5005

This is a complete refactor of the cache export Go interface.

Main aspect is to provide a strictly ordered walk of the
cache tree instead of previous one where modification could be
added to cache tree at any time by any component.

This should address subtle concurrency issues and remove
large parts of complicated (and likely buggy) normalization
and deduplication steps.

[BtbN]

I gave this a test on my setup that regularily runs into the cache issues, but now instead of spinning forever until timeout, those same jobs all fail with:

#193 exporting cache to registry
#193 preparing build cache for export
ERROR: failed to build: failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF

I guess that IS better than spinning forever though. I'll try deleting those questionable :cache tags from the registry next, and see how the builder behaves.

[tonistiigi]

@BtbN Is there a better error in the daemon logs? Looks like potentially a panic.

[BtbN]

It's on GitHub Actions, so unfortunately that log is all I got: https://github.com/BtbN/FFmpeg-Builds/actions/runs/17278597172/attempts/2

[tonistiigi]

@BtbN Can you rerun it with the debug checkbox checked?

[BtbN]

It's currently re-running without it. If it still fails, I'll do that.

[BtbN]

Unfortunately, me deleting all the affected :cache-tags from the registry made all jobs finish without issue (and also cleanly push new :cache tags).
My repo is prone to run into this though, so I'll make sure to remember to pull a debug log next time it happens.

[gamer191]

@tonistiigi here are debug logs from a fork of BtbN's repo. Hope these help!

https://github.com/yt-dlp/FFmpeg-Builds/actions/runs/17316380528/job/49168694925

https://github.com/yt-dlp/FFmpeg-Builds/actions/runs/17316380528/job/49168694929

(They're for 2 different builds, I'm not sure why one got so much further than the other)

[tonistiigi]

This is finally green.

@gamer191 @BtbN This was indeed a panic that was fixed in 8514fc1

If you have any issues with the cache exporting stage in complicated builds, please test this PR.

[BtbN]

I'll make the builds use an image built from this PR. They've been using a slightly older version from a few days ago (after I saw the first "fix panic" commit), and I have not noticed any failures or ultra-long stalls or timeouts since.

[cboggs]

This PR appears to fix (and will close once merged) #6008
See my repro and fix testing comments therein.

[crazy-max]

Very nice work @tonistiigi!

The normalization step before was hard to follow. Glad it got drastically reduced.

I also tested this PR on several projects locally and on GHA and good so far. I will make more tests using the new remote cache config debug #6076 (review).

[crazy-max]

Suggested change

	// make sure that none of the deps are childeren of r
	// make sure that none of the deps are children of r

[crazy-max]

I guess the bool returns true if the CacheLink slice is not complete. Could we have that as comment maybe?

[tonistiigi]

No, incomplete dependency is an error

buildkit/cache/remotecache/v1/chains.go

Line 74 in aed2e4a

return nil, false, errors.Errorf("empty dependency for %s", dgst)

[tonistiigi]

@jedevc @sipsma PTAL if there are any issues. There were also some lazy semantics fixes in here.