WCOW buildkit/buildx caching

[fruzitent]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

For some reason, specifically buildx, does not cache unpacked Windows Servercore image, but does Windows Nanoserver.
Buildkit, on the other hand, does cache stages from both images properly.

Steps to reproduce

1.1. Build nanoserver image using buildx
1.2. Build nanoserver image using buildx again (caching works)
1.3. Build servercore image using buildx
1.4. Build servercore image using buildx again (caching does not work)

2.1 Build nanoserver image using buildctl
2.2 Build nanoserver image using buildctl again (caching works)
2.3 Build servercore image using buildctl
2.4 Build servercore image using buildctl again (caching works)

3.1 Build nanoserver image using buildx (it loaded cache from buildkit node)
3.2 Build servercore image using buildx (it loaded cache from buildkit node)

Expected behaviour

Following builds to skip unchanged stages.

Actual behaviour

Buildx does not save stage cache, however, is able to import it (if there is one) from Buildkit.

Buildx version

github.com/docker/buildx v0.13.1-desktop.1 2368b2ee41873b948cb607251cec573ed00f00f9

Docker info

Client:
 Version:    26.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.26.1-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.27
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.23
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.6.3
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 5
 Server Version: 26.0.0
 Storage Driver: windowsfilter
  Windows:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local splunk syslog
 Swarm: inactive
 Default Isolation: hyperv
 Kernel Version: 10.0 22631 (22621.1.amd64fre.ni_release.220506-1250)
 Operating System: Microsoft Windows Version 23H2 (OS Build 22631.3447)
 OSType: windows
 Architecture: x86_64
 CPUs: 16
 Total Memory: 15.38GiB
 Name: DESKTOP-A7O2H12
 ID: e9984d83-abc3-4b05-a038-96db7debaeb5
 Docker Root Dir: C:\ProgramData\Docker
 Debug Mode: false
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Builders list

NAME/NODE             DRIVER/ENDPOINT                  STATUS     BUILDKIT   PLATFORMS
buildkit-exp*         remote
 \_ buildkit-exp0      \_ npipe:////./pipe/buildkitd   inactive
default               docker
 \_ default            \_ default                      running    v0.13.1    windows/amd64
desktop-linux         docker
 \_ desktop-linux      \_ desktop-linux                running    v0.13.1    linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
desktop-windows       docker
 \_ desktop-windows    \_ desktop-windows              running    v0.13.1    windows/amd64

Configuration

FROM mcr.microsoft.com/windows/nanoserver:ltsc2022

RUN <<EOF
    echo "Hello, nanoserver!"
EOF

FROM mcr.microsoft.com/windows/servercore:ltsc2022

RUN <<EOF
    echo "Hello, servercore!"
EOF

Build logs

No response

Additional info

There is a seemingly related issue on the buildkit repo moby/buildkit#2274.
I tried compiling buildkitd from the master after that fix was merged and repeating the same steps, but it did not change anything.

step1_buildkitd.log
step1_buildx.log
step1_containerd.log

step2_buildctl.log
step2_buildkitd.log
step2_containerd.log

step3_buildkitd.log
step3_buildx.log
step3_containerd.log

[tonistiigi]

cc @profnandaa

I wonder if this is the GC default storage limit kicking in and servercore is the bigger image. Check buildx inspect and buildx du.

[fruzitent]

➜ ~  docker buildx inspect
Name:          buildkit-exp
Driver:        remote
Last Activity: 2024-04-15 13:39:20 +0000 UTC

Nodes:
Name:             buildkit-exp0
Endpoint:         npipe:////./pipe/buildkitd
Status:           running
BuildKit version: v0.13.1
Platforms:        windows/amd64
Labels:
 org.mobyproject.buildkit.worker.containerd.namespace: buildkit
 org.mobyproject.buildkit.worker.containerd.uuid:      80aae2c4-eafe-43af-809c-2bbee660b401
 org.mobyproject.buildkit.worker.executor:             containerd
 org.mobyproject.buildkit.worker.hostname:             DESKTOP-A7O2H12
 org.mobyproject.buildkit.worker.network:              cni
 org.mobyproject.buildkit.worker.selinux.enabled:      false
 org.mobyproject.buildkit.worker.snapshotter:          windows
GC Policy rule#0:
 All:           false
 Filters:       type==source.local,type==exec.cachemount,type==source.git.checkout
 Keep Duration: 48h0m0s
 Keep Bytes:    488.3MiB
GC Policy rule#1:
 All:           false
 Keep Duration: 1440h0m0s
 Keep Bytes:    1.863GiB
GC Policy rule#2:
 All:        false
 Keep Bytes: 1.863GiB
GC Policy rule#3:
 All:        true
 Keep Bytes: 1.863GiB
➜ ~  docker buildx du
ID                                              RECLAIMABLE     SIZE            LAST ACCESSED
fozyq86h7n81d4gr68b00jkjb                       true            18.01GB*        9 hours ago
2ayz1tef4e4bq7lyoqadlox0d                       true            17.98GB*        7 hours ago
ifkou0cwokqo6zgea4jlqmuzt                       true            7.336GB*        11 hours ago
7vvwuu919c9ffzy4ot852qxmo                       true            5.941GB*        11 hours ago
mtxdriimlm219j2s6tynkm78w                       true            939.6MB*        7 hours ago
cq9f7xsqzf50nemwfzje7ugs2                       true            746.8MB         11 hours ago
xxv5vxg8bur524vwnglgwudgf                       true            673.5MB         11 hours ago
o2a1dzv5jasbiqpfnw8dy6zfy                       true            174.9MB         11 hours ago
ns32won5iib26msan08j9xs4w*                      true            4.933kB         7 hours ago
n4qs13u4o8k7650v9fxm57gnr*                      true            101B            11 hours ago
n6vytn04tfgpb41ki2rd1mrcc*                      true            0B              7 hours ago
hg2roawofuky4ch58ty2mbhng*                      true            0B              11 hours ago
Shared:         50.21GB
Private:        1.595GB
Reclaimable:    51.81GB
Total:          51.81GB

[tonistiigi]

GC configured at 1.8 GB is probably not what you want with such big images. Set --containerd-worker-gc false to turn off the GC (or keepstorage to tweak it).

@profnandaa We need to look into how such a small limit happened. And maybe set a minimum for wcow as it tends to use huge images.

[fruzitent]

(unrelated) Docker's Desktop built-in runner (does not support buildkit) says it has unlimited storage

[profnandaa]

Sure, will take a look. Thanks! ./na

…

On Tue, Apr 16, 2024, 03:36 Tõnis Tiigi ***@***.***> wrote: GC configured at 1.8 GB is probably not what you want with such big images. Set --containerd-worker-gc false to turn off the GC (or keepstorage to tweak it). @profnandaa <https://github.com/profnandaa> We need to look into how such a small limit happened. And maybe set a minimum for wcow as it tends to use huge images. — Reply to this email directly, view it on GitHub <#2411 (comment)> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB7ZEPI57U6VD6NJWPKHSLY5RW7BBFKMF2HI4TJMJ2XIZLTSWBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVIZDGMJZGQYDOOBZG6SG4YLNMWUWQYLTL5WGCYTFNSBKK5TBNR2WLKRUGUZDQOBZGIYDAMFENZQW2ZNJNBQXGX3MMFRGK3FMON2WE2TFMN2F65DZOBS2YSLTON2WKQ3PNVWWK3TUUZ2G64DJMNZZJAVEOR4XAZNKOJSXA33TNF2G64TZUV3GC3DVMWUTCNZXGIYTANRSG6BKI5DZOBS2K2LTON2WLJLWMFWHKZNKGIZDIMZYGE3TCNJXQKSHI6LQMWSWYYLCMVWKK5TBNR2WLKRSGMYTSNBQG44DSN4CUR2HS4DFUVWGCYTFNSSXMYLMOVS2UNBVGI4DQOJSGAYDBJ3UOJUWOZ3FOKTGG4TFMF2GK> . You are receiving this email because you were mentioned. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .

[tonistiigi]

Docker's Desktop built-in runner (does not support buildkit) says it has unlimited storage

That's not what you are using to build. Based on your other output the builder you are using is called buildkit-exp

[fruzitent]

tested with buildkitd --containerd-worker-gc=false it works

➜ wcow-test  ✗ docker buildx inspect
Name:          buildkit-exp
Driver:        remote
Last Activity: 2024-04-16 03:59:55 +0000 UTC

Nodes:
Name:             buildkit-exp0
Endpoint:         npipe:////./pipe/buildkitd
Status:           running
BuildKit version: v0.13.1
Platforms:        windows/amd64
Labels:
 org.mobyproject.buildkit.worker.containerd.namespace: buildkit
 org.mobyproject.buildkit.worker.containerd.uuid:      b84d108d-a0e9-4a2e-a687-006295e0a86f
 org.mobyproject.buildkit.worker.executor:             containerd
 org.mobyproject.buildkit.worker.hostname:             DESKTOP-A7O2H12
 org.mobyproject.buildkit.worker.network:              cni
 org.mobyproject.buildkit.worker.selinux.enabled:      false
 org.mobyproject.buildkit.worker.snapshotter:          windows
➜ wcow-test  docker buildx du
ID                                              RECLAIMABLE     SIZE            LAST ACCESSED
af3fw2gwc53dxsd89n8s2sfrl                       true            7.336GB         14 minutes ago
sax8i4cwg8z1lpjqgrbvpajmg                       true            5.941GB         2 minutes ago
n2n3th25eybkblxaj6dn1vtay                       true            746.7MB         32 seconds ago
m1rgmim2liy644g4g4122gi6c*                      true            97B             32 seconds ago
i6dkls59gv6qviaapskbsdckz*                      true            0B              32 seconds ago
Reclaimable:    14.02GB
Total:          14.02GB

[tonistiigi]

@profnandaa I think I found the issue.

If no GC is configured by the user then in Linux the default is 10% of disk: https://github.com/moby/buildkit/blob/master/cmd/buildkitd/config/gcpolicy_unix.go#L11

But on wcow it is just a constant 2GB 😬 https://github.com/moby/buildkit/blob/master/cmd/buildkitd/config/gcpolicy_windows.go#L7

[profnandaa]

@tonistiigi -- thanks! You think we should just do the same for WCOW for consistency? I can work on the PR.

[thompson-shaun]

Let's transfer this to moby/buildkit

[tonistiigi]

moby/buildkit#4858