nerdctl image save fails when discard_unpacked_layers=true is enabled

[2rigor]

Description

When GC layers optimization is enabled (discard_unpacked_layers=true) some/all layers may be missing from containerd content storage.
Thus nerdctl image save fails to export image. Can CLI be fixed to export image content? Practically the content is present on the disk

Similarly to containerd/containerd#9277

Steps to reproduce the issue

1. Install K8s with containerd
2. Download and install nerdctl (from github)
3. Enable discard_unpacked_layers=true in /etc/containerd/config.toml
   Alternatively you may create e.g. GKE 1.27
4. list images using sudo nerdctl -n k8s.io images, choose some image. Instead run pod with some image e.g. alpine:latest
   sudo nerdctl -n k8s.io image save <image_ref> > /dev/null e.g. sudo nerdctl -n k8s.io image save docker.io/library/alpine:latest > /dev/null

Describe the results you received and expected

You will get error:
FATA[0000] failed to get reader: content digest sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17: not found

Expected - no failure (and to get tar file)

What version of nerdctl are you using?

Client:
Version: v1.6.2
OS/Arch: linux/amd64
Git commit: e3dc23b
buildctl:
Version:

Server:
containerd:
Version: 1.6.24
GitCommit: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
runc:
Version: 1.1.9
GitCommit: v1.1.9-0-gccaecfc

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

Client:
Namespace: default
Debug Mode: false

Server:
Server Version: 1.6.24
Storage Driver: overlayfs
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Log: fluentd journald json-file syslog
Storage: native overlayfs
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.15.0-84-generic
Operating System: Ubuntu 22.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 4.798GiB
Name: rigor-ctrd-42
ID: abe6dcae-815f-4437-845d-09e1ef8851ee

[dineshp-09]

Any updates on this? The issue is reproducible on GKE version 1.27 and above and EKS version 1.26 and above.

[2rigor]

Any updates on this? The issue is reproducible on GKE version 1.27 and above and EKS version 1.26 and above.

In fact, @dineshp-09, based on the below it should probably hannep on any support EKS with recent AMIs (since Jul-28 this year): https://github.com/awslabs/amazon-eks-ami/blob/5b54e37cde1e8a4c67a3ddaff27c953fc7b7b37f/CHANGELOG.md?plain=1#L270

[apostasie]

@2rigor we (recently) merged a slew of fixes related to the specific problem of missing layers, including for save, especially for Kube.

If this is still a concern for you, would you be able to give it a spin with the latest main nerdctl? (or at least the latest v2.0.x released version)

Alternatively, I believe we should close this as fixed.

cc @AkihiroSuda

[2rigor]

Thanks for the update @apostasie! So, image save should work now even when discard_unpacked_layers=true?
We have implemented some workaround based on snapshots. I don't think I will get to test the latest main nerdctl soon. If you prefer - the issue can be closed.

[apostasie]

Thanks for the update @apostasie! So, image save should work now even when discard_unpacked_layers=true?

We need confirmation for that specific case, though we did fix a slew of image save issues with missing unpacked layers.

We have implemented some workaround based on snapshots. I don't think I will get to test the latest main nerdctl soon. If you prefer - the issue can be closed.

Ok, I will give it a spin on a kind cluster and let you know.
Alternatively, we can just keep this around until the next patch release so that you can confirm in your specific environment?

[2rigor]

Since I'm working on a different project, I'm not sure I could spend some time confirming this case when the next patch release is out. I'm ok with whatever fits you regarding this issue status

…

On Sun, Apr 27, 2025, 21:06 Apostasie ***@***.***> wrote: *apostasie* left a comment (containerd/nerdctl#2588) <#2588 (comment)> Thanks for the update @apostasie <https://github.com/apostasie>! So, *image save* should work now even when discard_unpacked_layers=true? We need confirmation for that specific case, though we did fix a slew of image save issues with missing unpacked layers. We have implemented some workaround based on snapshots. I don't think I will get to test the latest main nerdctl soon. If you prefer - the issue can be closed. Ok, I will give it a spin on a kind cluster and let you know. Alternatively, we can just keep this around until the next patch release so that you can confirm in your specific environment? — Reply to this email directly, view it on GitHub <#2588 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJJZ6XVKI2GPPLEBWFQ6W6D23UMDFAVCNFSM6AAAAAB3NN2WM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMZTGU3TINRRGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[apostasie]

@2rigor all good - I'll just confirm in a CRI env on my side and close if needed.
Cheers!