Skip to content

Conversation

@weng271190436
Copy link
Contributor

@weng271190436 weng271190436 commented Nov 4, 2025

Description of your changes

Was hitting no space left on device in the backport PR e2e tests

Command Output: ctr: failed to extract layer (application/vnd.oci.image.layer.v1.tar sha256:97db83035163d79cf9ecf009c859d2cf93b4d11307f28ad7526beddd84cd1da1) to overlayfs as "extract-771242434-d36z sha256:34af12c2cba687b2ab04efbed891a15eb9adfb188acc24bcd46f25dc44f79f3d": write /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/15/fs/crdinstaller: no space left on device

Inspected the storage usage in the github action runner and realized that we can use /dev/sda1 (which is mounted to /mnt) for docker.

=== Disk usage after loading images into cluster-2 cluster ===
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        72G   71G  1.4G  99% /
tmpfs           7.9G   84K  7.9G   1% /dev/shm
tmpfs           3.2G  2.2M  3.2G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sdb16      881M   62M  758M   8% /boot
/dev/sdb15      105M  6.2M   99M   6% /boot/efi
/dev/sda1        74G  4.1G   66G   6% /mnt
tmpfs           1.6G   12K  1.6G   1% /run/user/1001

The same fix is not applicable to upstream kubefleet repo because upstream CI machine has a big /dev/root with no secondary storage device

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       581G   71G  510G  13% /
tmpfs            32G   84K   32G   1% /dev/shm
tmpfs            13G  2.2M   13G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sda16      881M   62M  758M   8% /boot
/dev/sda15      105M  6.2M   99M   6% /boot/efi
tmpfs           6.3G   12K  6.3G   1% /run/user/1001
tmpfs           6.3G   12K  6.3G   1% /run/user/0

I have:

  • Run make reviewable to ensure this PR is ready for review.

How has this code been tested

Special notes for your reviewer

@weng271190436 weng271190436 changed the title fix: move docker data dir to /mnt to address e2e test low disk space fix: move docker data dir to /mnt to address e2e test low disk space for clean backport Nov 4, 2025
Signed-off-by: Wei Weng <Wei.Weng@microsoft.com>
@weng271190436 weng271190436 force-pushed the weiweng/use-secondary-storage-for-e2e branch from a57fb3b to 5e65a52 Compare November 4, 2025 15:15
@weng271190436 weng271190436 merged commit 0b60f33 into Azure:main Nov 7, 2025
23 of 24 checks passed
@weng271190436 weng271190436 deleted the weiweng/use-secondary-storage-for-e2e branch November 7, 2025 03:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants