debug: allow debug from remote cache config

[tonistiigi]

follow-up to #6061

Allows debugging from remote config passed via POST request in addition to debugging local cache store.

[tonistiigi]

Leaving draft for now as there seems to be an extra level of indirection when debugging remote source, that comes from output index being embedded into the cache key. I was aware of output index being stored as a separate value only in bolt but bit unexpected how these behave differently even in the CacheKeyStorage interface level that I want to investigate more.

[tonistiigi]

Leaving draft for now as there seems to be an extra level of indirection when debugging remote source, that comes from output index being embedded into the cache key. I was aware of output index being stored as a separate value only in bolt but bit unexpected how these behave differently even in the CacheKeyStorage interface level that I want to investigate more.

In order to fix this, I added the same indirection to the boltdb debug as well, so that both backends generate matching keys.

Also added extra commit for better output:

@tonistiigi
debug: combine links from parents intead of children
80e6c06
Grouping the link digests to parent side (links pointing
to current node) makes more sense than children side (links
pointing to next nodes). This also reduces the duplicates
in the output as previously same digest link could need
plaintext under multiple nodes.

[crazy-max]

With the dev builder from our compose stack with slight changes to save debug cache:

name: buildkit
services:
  buildkit:
    container_name: buildkit-dev
    build:
      context: ../..
    image: moby/buildkit:local
    command: --save-cache-debug
    ports:
      - 127.0.0.1:5000:5000
      - 127.0.0.1:6060:6060
    restart: always
    privileged: true
    configs:
      - source: buildkit_config
        target: /etc/buildkit/buildkitd.toml
    volumes:
      - buildkit:/var/lib/buildkit

volumes:
  buildkit:

configs:
  buildkit_config:
    file: ./buildkitd.toml

Create Buildx dev builder:

$ docker buildx create \
    --bootstrap \
    --name dev \
    --driver remote \
    docker-container://buildkit-dev

$ docker buildx inspect dev
Name:          dev
Driver:        remote
Last Activity: 2025-08-12 08:22:48 +0000 UTC

Nodes:
Name:             dev0
Endpoint:         docker-container://buildkit-dev
Status:           running
BuildKit version: v0.23.0-rc1-71-g854eace66.m
Platforms:        linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
Labels:
 org.mobyproject.buildkit.worker.executor:         oci
 org.mobyproject.buildkit.worker.hostname:         9d52fb535535
 org.mobyproject.buildkit.worker.network:          host
 org.mobyproject.buildkit.worker.oci.process-mode: sandbox
 org.mobyproject.buildkit.worker.selinux.enabled:  false
 org.mobyproject.buildkit.worker.snapshotter:      overlayfs
GC Policy rule#0:
 All:            false
 Filters:        type==source.local,type==exec.cachemount,type==source.git.checkout
 Keep Duration:  48h0m0s
 Max Used Space: 488.3MiB
GC Policy rule#1:
 All:            false
 Keep Duration:  1440h0m0s
 Reserved Space: 9.313GiB
 Max Used Space: 93.13GiB
 Min Free Space: 188.1GiB
GC Policy rule#2:
 All:            false
 Reserved Space: 9.313GiB
 Max Used Space: 93.13GiB
 Min Free Space: 188.1GiB
GC Policy rule#3:
 All:            true
 Reserved Space: 9.313GiB
 Max Used Space: 93.13GiB
 Min Free Space: 188.1GiB

For this Dockerfile:

FROM alpine                                                                                                                                                                                                    
RUN echo hello                                                                                                                                                                                                 
EXPOSE 8080-8085

And exporting cache to registry:

docker buildx --builder builder build --cache-to=type=registry,ref=crazymax/test:buildkit-6076 .

Then send POST request of application/vnd.buildkit.cacheconfig.v0 from https://explore.ggcr.dev/?image=crazymax%2Ftest%3Abuildkit-6076

I get with the new format:

ID: 1
Digest: sha256:89613c014914fc647717af0c3feecb02af98ec64275ff68909a59c3c8e35fd0f
Children:
  Input 0:	 2
Plaintexts:
sha256:89613c014914fc647717af0c3feecb02af98ec64275ff68909a59c3c8e35fd0f (string):
  sha256:8c72bc153f13bdf1c78dd16a56bad285b50014c3d58b61ae296de88d5397eebe (string):
  data: sha256:8c72bc153f13bdf1c78dd16a56bad285b50014c3d58b61ae296de88d5397eebe@0

sha256:8c72bc153f13bdf1c78dd16a56bad285b50014c3d58b61ae296de88d5397eebe (string):
  data: buildkit.source.v0:sha256:418dccb7d85a63a6aa574439840f7a6fa6fd2321b3e2394568a317735e867d35


ID: 2
Parents:
  Input 0:	1	sha256:cc01ae207aafb5c2c02bdb521a92039d3ee0f1ca84075b1a9307a50316c7b353
    Selector: sha256:8a5edab282632443219e051e4ade2d1d5bbc671c781051bf1437897cbdfea0f1
  Input 0:	3	sha256:cc01ae207aafb5c2c02bdb521a92039d3ee0f1ca84075b1a9307a50316c7b353
    Selector: sha256:8a5edab282632443219e051e4ade2d1d5bbc671c781051bf1437897cbdfea0f1
Plaintexts:
sha256:cc01ae207aafb5c2c02bdb521a92039d3ee0f1ca84075b1a9307a50316c7b353 (string):
  sha256:6eeaec2f161bb1b76627557b14a4526acfdd07624d6030206e2f9215abf8bcbe (json):
  data: sha256:6eeaec2f161bb1b76627557b14a4526acfdd07624d6030206e2f9215abf8bcbe@0

sha256:6eeaec2f161bb1b76627557b14a4526acfdd07624d6030206e2f9215abf8bcbe (json):
  data: {"Type":"buildkit.exec.v0","Exec":{"meta":{"args":["/bin/sh","-c","echo hello"],"env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"cwd":"/","removeMountStubsRecursive":true}},"OS":"linux","Arch":"amd64"}

sha256:8a5edab282632443219e051e4ade2d1d5bbc671c781051bf1437897cbdfea0f1 (string):
  data: /


ID: 3
Digest: sha256:8e06a68b6fc81bc311289cc094a7c7162061e58d2dbc0f6baa8df5f7e14443c5
Children:
  Input 0:	 2
Plaintexts:
sha256:8e06a68b6fc81bc311289cc094a7c7162061e58d2dbc0f6baa8df5f7e14443c5 (string):
  sha256:a8c45412a3163e7c61cb2dd6e42376c41ec924543f7eec0899cb494b3d7f5e71 (string):
  data: sha256:a8c45412a3163e7c61cb2dd6e42376c41ec924543f7eec0899cb494b3d7f5e71@0

sha256:a8c45412a3163e7c61cb2dd6e42376c41ec924543f7eec0899cb494b3d7f5e71 (string):
  data: buildkit.source.v0:sha256:95d4982c89fd6816afdab744a3b6513c9072f870a434e95f325ba0aa1c77c47f

[crazy-max]

Wonder if we could have a /debug/cache/import endpoint that could use the same CSV attributes as cache-from to make this easier? Like type=registry,ref=foo/bar:cache

[tonistiigi]

This can't be easily done like this as things like authentication require buildkit features like grpc session tunnel and this is a completely different debug API endpoint. I think the way UX can work eventually is that something like buildx can fetch the config on the client side and then upload to the debug endpoint.