c8d/push: Push lazy blobs with distribution source labels

[vvoland]

This makes it possible to push multi-platform images if some contents are missing from the local content store, but can be fetched from a source repository.

The trick here is wrapping the ContentStore to return a fake content.Info with a containerd.io/distribution.source. label which specifies the source registry and repository for the blob. It's then used by the repository to cross-repo mount the blobs.

Caveats:

• Doesn't work with other registries than the one which provided the missing blob.
• Works only as long as we have a source manifest (it's deleted by buildx purge -a or restarting the daemon)

Test:

printf 'FROM alpine\nCMD ["echo", "hello"]' | docker buildx build . \
    -t docker.io/repo/test:latest --platform linux/amd64,linux/arm64
docker push docker.io/repo/test:latest

Test result with a fresh repo

root # docker buildx imagetools inspect docker.io/pawelgronowski465/pushworks:220824 # Repo doesn't exist1
ERROR: docker.io/pawelgronowski465/pushnotworks:220824: not found
root # cat Dockerfile
FROM alpine
CMD ["echo", "test"]
root # docker build . -t pawelgronowski465/pushworks:220824 --platform linux/arm64,linux/amd64,linux/386
[+] Building 4.4s (9/9) FINISHED
 => [internal] load .dockerignore                                                                                                         0.0s
 => => transferring context: 2B                                                                                                           0.0s
 => [internal] load build definition from Dockerfile                                                                                      0.0s
 => => transferring dockerfile: 70B                                                                                                       0.0s
 => [linux/arm64 internal] load metadata for docker.io/library/alpine:latest                                                              3.9s
 => [linux/amd64 internal] load metadata for docker.io/library/alpine:latest                                                              3.9s
 => [linux/386 internal] load metadata for docker.io/library/alpine:latest                                                                3.9s
 => [linux/amd64 1/1] FROM docker.io/library/alpine@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad               0.0s
 => => resolve docker.io/library/alpine@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad                           0.0s
 => [linux/386 1/1] FROM docker.io/library/alpine@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad                 0.0s
 => => resolve docker.io/library/alpine@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad                           0.0s
 => [linux/arm64 1/1] FROM docker.io/library/alpine@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad               0.4s
 => => resolve docker.io/library/alpine@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad                           0.0s
 => => sha256:9b18e9b68314027565b90ff6189d65942c0f7986da80df008b8431276885218e 0B / 2.71MB                                                0.4s
 => exporting to image                                                                                                                    0.4s
 => => exporting layers                                                                                                                   0.0s
 => => exporting manifest sha256:1e3431ac147d5ab9aad7408f9e9cf77191b909b8d5eeee4c3074d29e012a7fcf                                         0.0s
 => => exporting config sha256:03a88cfd93db02177bb2b47b7fc73504ebf75b7cc3eb4432edc01caae5535098                                           0.0s
 => => exporting manifest sha256:89aebd05109a4dba92ff76ea73b52ded8286b2c5a30467c9810eb3c6b92db91a                                         0.0s
 => => exporting config sha256:9faa6ecf5f1b46d07c961b9a490a16391a9b1f7fb30cea316dbe10992595207a                                           0.0s
 => => exporting manifest sha256:c8ed6a8c58f8e50c90be744426398f3b2fcbea51d45f0e27a266b1a69cf99d5c                                         0.0s
 => => exporting config sha256:9cff1119afd2277751867756a94fe0f20495b47374cc84d8fe7096006cdf272e                                           0.0s
 => => exporting manifest list sha256:610e37756255c286b3e3706d3239e339f4b378487c4aafe03c0941cae65ff462                                    0.0s
 => => naming to docker.io/pawelgronowski465/pushworks:220824                                                                             0.0s
 => => unpacking to docker.io/pawelgronowski465/pushworks:220824                                                                          0.4s
root # docker push docker.io/pawelgronowski465/pushworks:220824
03a88cfd93db: Pushed
9b18e9b68314: Pushed
89aebd05109a: Pushed
c8ed6a8c58f8: Pushed
610e37756255: Pushed
9cff1119afd2: Pushed
6c0d3b419d84: Pushed
213ec9aee27d: Pushed
1e3431ac147d: Pushed
9faa6ecf5f1b: Pushed
root # docker buildx imagetools inspect docker.io/pawelgronowski465/pushworks:220824
Name:      docker.io/pawelgronowski465/pushworks:220824
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:610e37756255c286b3e3706d3239e339f4b378487c4aafe03c0941cae65ff462

Manifests:
  Name:      docker.io/pawelgronowski465/pushworks:220824@sha256:1e3431ac147d5ab9aad7408f9e9cf77191b909b8d5eeee4c3074d29e012a7fcf
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64

  Name:      docker.io/pawelgronowski465/pushworks:220824@sha256:89aebd05109a4dba92ff76ea73b52ded8286b2c5a30467c9810eb3c6b92db91a
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64

  Name:      docker.io/pawelgronowski465/pushworks:220824@sha256:c8ed6a8c58f8e50c90be744426398f3b2fcbea51d45f0e27a266b1a69cf99d5c
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/386

[vvoland]

Undrafting this as it allows the build+push case to work.
Will work on the pull+tag+push case separately.

[vvoland]

Added some comments and cleaned up the source label handling a bit.
Will squash the commits later when merging.

[thaJeztah]

ah, have a call, so need to continue later; submitted some nits (no blockers so far)

[thaJeztah]

Suggested change

	// Do a small peek of the content to check if content is definitely not a json.
	// Returns true when content is definitely not a json.
	// If it returns false, then the content may still not be a json.
	// peekNotJson does a small peek of the content to check if content is definitely not JSON.
	// It returns true if content is definitely not JSON, or false if it was unable to detect if it's
	// JSON or not.

The error doesn't not appear to be described (on how it should be used in combination with the bool)

[vvoland]

Thanks. I didn't bother much describing the error case, as it's not really something that should happen under normal circumstances. This is called only when traversing the content store, so we expect the content to be there.
The only case of error here would be if the content is silently deleted in background between the beginning of the Walk and running handler for this particular content.
BTW, I wonder if there is some way to "lock" the store, so we can be 100% sure that this won't happen.

[rumpl]

I think you can use a leased context maybe?

[vvoland]

But how do I put a lease on everything in the Content store? :P
And if I do - will lease be enough to pause something like ctr content remove <hash> until the Walk is finished?

[rumpl]

Yeah nah it doesn't work, the lease you add to your context is added to objects you create during that context.

[thaJeztah]

Left some comments, but mostly looks sane 😄

[thaJeztah]

LGTM, thanks!

Let's make sure we have a tracking ticket to improve and upstream a more official approach, and discuss with the containerd maintainers.

[thaJeztah]

When upstreaming:

• depends on / needs to go after containerd integration: produce progress events polling ctrd's content.Store moby/moby#43819
• but, we may want to squash all fixes related to "common" progress (push/pull) code into a single commit

combine with one or more of;

• containerd: Implement push #15
• Push progress #16
• c8d/progress: Fix progress not ending #73
• c8d/push: Push lazy blobs with distribution source labels #74
• progress fix-up from c8d/push: Fetch missing resources that can't be mounted #80