The `nerdctl save` command pulls the image when the image is already pulled by the `ctr i pull`.

[yankay]

Description

The nerdctl save command pulls the image when the image is already pulled by the ctr i pull.
And the nerdctl save should not download the image.

Steps to reproduce the issue

1. Clean the Environment

service containerd stop
rm -rf /var/lib/containerd/*
service containerd start

2. Pull image with ctr

(base) root@kay201:~# ctr -n k8s.io images pull m.daocloud.io/registry.k8s.io/pause:3.10 # You can change to registry.k8s.io/pause:3.10
m.daocloud.io/registry.k8s.io/pause:3.10 saved
└──index (ee6521f290b2) complete |++++++++++++++++++++++++++++++++++++++|
├──manifest (e50b7059b633) complete |++++++++++++++++++++++++++++++++++++++|
│ └──config (afb61768ce38) complete |++++++++++++++++++++++++++++++++++++++|
├──manifest (7c38f24774e3) complete |++++++++++++++++++++++++++++++++++++++|
│ ├──config (873ed7510279) complete |++++++++++++++++++++++++++++++++++++++|
│ └──layer (61d9e957431b) complete |++++++++++++++++++++++++++++++++++++++|
├──manifest (8a5fb194104a) complete |++++++++++++++++++++++++++++++++++++++|
│ └──config (9d01e62aca03) complete |++++++++++++++++++++++++++++++++++++++|
├──manifest (41ff8c30d655) complete |++++++++++++++++++++++++++++++++++++++|
│ └──config (3f721004d67b) complete |++++++++++++++++++++++++++++++++++++++|
├──manifest (89e51d7a4cb6) complete |++++++++++++++++++++++++++++++++++++++|
│ └──config (b2535becc394) complete |++++++++++++++++++++++++++++++++++++++|
├──manifest (e3ecdc4b6bd3) complete |++++++++++++++++++++++++++++++++++++++|
│ └──config (cddd0dd4df88) complete |++++++++++++++++++++++++++++++++++++++|
└──manifest (193fcedcfac4) complete |++++++++++++++++++++++++++++++++++++++|
└──config (4bea0b366d20) waiting |--------------------------------------|
application/vnd.docker.distribution.manifest.list.v2+json sha256:ee6521f290b2168b6e0935a181d4cff9be1ac3f505666ef0e3c98fae8199917a
Pulling from OCI Registry (m.daocloud.io/registry.k8s.io/pause:3.10) elapsed: 3.6 s total: 323.9 (91.0 KiB/s)

3. Save the image with nerdctl
   It began to pull the images and save it.

(base) root@kay201:~# nerdctl -n k8s.io image save -o /tmp/registry.k8s.io_pause_3.10.tar m.daocloud.io/registry.k8s.io/pause:3.10
m.daocloud.io/registry.k8s.io/pause:3.10: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:ee6521f290b2168b6e0935a181d4cff9be1ac3f505666ef0e3c98fae8199917a: exists |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:e50b7059b633caf3c1449b8da680d11845cda4506b513ee7a2de00725f0a34a7: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:75e060e453aa927883755f715daa02fb335ea7f148a8ab249f779be796d4bb7e: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:afb61768ce381961ca0beff95337601f29dc70ff3ed14e5e4b3e5699057e6aa8: exists |++++++++++++++++++++++++++++++++++++++|
elapsed: 1.0 s total: 0.0 B (0.0 B/s)
m.daocloud.io/registry.k8s.io/pause:3.10: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:ee6521f290b2168b6e0935a181d4cff9be1ac3f505666ef0e3c98fae8199917a: exists |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:41ff8c30d6555f5dd6991884d258d7c24b9aeee2d2b75eaa24391f9c702da04b: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:07829728559256c8924ddb7a4b5407680a876f85ca26138139a5d456051c7d7c: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:3f721004d67b32297dffb2dff69b60a7e03ef2c16da474dfc8d97dcfe8e3e992: exists |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.8 s total: 0.0 B (0.0 B/s)
m.daocloud.io/registry.k8s.io/pause:3.10: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:ee6521f290b2168b6e0935a181d4cff9be1ac3f505666ef0e3c98fae8199917a: exists |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:89e51d7a4cb6a0436b23d34d28ee4ed16721a6853c4b76177c552d7d0ca4b8a7: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:a6203acf0eb69122886f8df6b0ab288b1d7b5d7e3ccef4e1e8740cd3ba390efd: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:b2535becc394bb04d6f0bda13f66be6e0338406eb258ec0584d8ece59eebef83: exists |++++++++++++++++++++++++++++++++++++++|
elapsed: 1.8 s total: 0.0 B (0.0 B/s)
m.daocloud.io/registry.k8s.io/pause:3.10: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:ee6521f290b2168b6e0935a181d4cff9be1ac3f505666ef0e3c98fae8199917a: exists |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:193fcedcfac410b3484b96f06c8a2cf750bcdb60b44b914798ec74b0ab8fb730: exists |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:e3ecdc4b6bd38ce47123195d566292581080abbb51b80603a9168f84fef0d1a2: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:26085ddda8400952278f21eda4de3abb08e6e2b02afaddd77d1dab2b2a3f7c70: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:3386ea7089240af044414bade0c265ed64590c09451104a81f643e90a0ddf72d: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:4bea0b366d2096fdb7eedccd0a067f1cb16b7c2c614dcc86dc7569de44b66ed0: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:51433a8bc864e113031f56050243d96f07ee45ee87cfb769f94878f710fcfc00: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:b3afe2cc66044efd5b50a5e5989b4c04950779145810e80983be60a4fd7f43e1: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:61b9d3e54fdf80b25126caf3898b7619f2cc4271b3707e147145c6aa7d7b6663: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:ac0271ec295db8efe49fd6d64caa24b7612828d8de3596df8118085a7f9c9f77: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:2a2a37705371efd292777b63ae5500db8c667ea6a2269db5a82bd7ddb37c611d: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:0a1660a7fc0943f082ad82712f70f1f9864c10fcd7b6c2aa34b66e320d28df50: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:a0d44e5352dcb84bca48b6ee3d30a9ec91b5e6eb6793747e06d2454d360a9338: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:cddd0dd4df88179fc1c6de0c8dd37af6660513a8d6dd713bf707768f74c32810: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:5ad177daa048ca8b354b9ad03deac863ff519a2860a35dc9fdc0011619aacc3c: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:543bb037d9827e706ea0ee9277e56ff916439a114fa56c520ac7dcaf6daae84a: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:efd3b1563a816d85c6414e0c139691df720c34d6f65abaa19819d37b11459b40: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:bc30bde5a6578b9643d05dd47105414777adadaf5df93b493eff1785e1e07328: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:77e7191206a99af5cf1718885fb45262c2e2da30ad650c5868dfa3c54739c24a: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:1d3139ed5a48fc73d591768cf518b1103bdab55173ff5a42f24fc45035f8a6c1: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:2b0c17d061d115d5a13d2328efb325147a8bfa52e0183515c85776f8cc72734d: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 24.3s total: 7.5 Mi (316.9 KiB/s)
m.daocloud.io/registry.k8s.io/pause:3.10: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:ee6521f290b2168b6e0935a181d4cff9be1ac3f505666ef0e3c98fae8199917a: exists |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:8a5fb194104a55b306eecde523eeebcb12d46d7e4990fc9c1a78f8265efc1ced: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:76f284805ee87bcb4a15601efff3dc1577034453d48965c80b3c3a9e51675a2e: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:9d01e62aca03869527adbcbf895cdff3745216bb84b489638d479e0d6a02478e: exists |++++++++++++++++++++++++++++++++++++++|
elapsed: 2.0 s total: 0.0 B (0.0 B/s)

Describe the results you received and expected

The nerdctl should not pull the image in the save command.

What version of nerdctl are you using?

Client:
Version: v2.0.3
OS/Arch: linux/amd64
Git commit: ce61fc4
buildctl:
Version: v0.18.1
GitCommit: eb68885955169461d72dc2b7e6d084100fcaba86

Server:
containerd:
Version: v2.0.0
GitCommit: 207ad711eabd375a01713109a8a197d197ff6542
runc:
Version: 1.2.2
GitCommit: v1.2.2-0-g7cb36325

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

Client:
Version: v2.0.3
OS/Arch: linux/amd64
Git commit: ce61fc4
buildctl:
Version: v0.18.1
GitCommit: eb68885955169461d72dc2b7e6d084100fcaba86

Server:
containerd:
Version: v2.0.0
GitCommit: 207ad711eabd375a01713109a8a197d197ff6542
runc:
Version: 1.2.2
GitCommit: v1.2.2-0-g7cb36325
(base) root@kay201:~# nerdctl info
Client:
Namespace: k8s.io
Debug Mode: false

Server:
Server Version: v2.0.0
Storage Driver: overlayfs
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Log: fluentd journald json-file none syslog
Storage: native overlayfs
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.15.0-125-generic
Operating System: Ubuntu 22.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.61GiB
Name: kay201
ID: a1caf8e2-cd9e-41c4-8ed4-7107aac5c2e8

[yankay]

It maybe imported by #3435 .

[apostasie]

Nerdctl will only do that when layers are missing: https://github.com/containerd/nerdctl/pull/3435/files#diff-75afba2e91ee649836246b829d7e6cc736da7c807884291945f9aea21517ad5eR78-R83

Can you confirm with ctr inspect of the target image that layers are indeed missing in that case?

[apostasie]

@yankay were you able to confirm the above?

[yankay]

@yankay were you able to confirm the above?

Hi @apostasie,

I encountered an issue with architecture-specific layer pulls using ctr on an x86 platform. When pulling the pause:3.10 image with the command:

ctr -n k8s.io images pull m.daocloud.io/registry.k8s.io/pause:3.10

It fetches all architecture manifests but does not download the layers for architectures other than x86. For instance, the arm64 layer '75e060e453a' is missing.

However, when inspecting the image with:

ctr -n k8s.io i inspect m.daocloud.io/registry.k8s.io/pause:3.10 | grep -C 2 75e060e453a

The output shows the layer as present in the inspection:

    │   │    Platform: linux/arm64
    │   ├── application/vnd.docker.container.image.v1+json @sha256:afb61768ce381961ca0beff95337601f29dc70ff3ed14e5e4b3e5699057e6aa8 (886 bytes)
    │   └── application/vnd.docker.image.rootfs.diff.tar.gzip @sha256:75e060e453aa927883755f715daa02fb335ea7f148a8ab249f779be796d4bb7e (264141 bytes)
    ├── application/vnd.docker.distribution.manifest.v2+json @sha256:41ff8c30d6555f5dd6991884d258d7c24b9aeee2d2b75eaa24391f9c702da04b (501 bytes)
    │   │    Platform: linux/ppc64le

Thanks.

[yankay]

After successfully pulling the image with nerdctl pull and then using nerdctl save, the process works as expected.
Therefore, this can be considered an unusual behavior.

[apostasie]

Got it, thanks a lot @yankay.

Indeed, EnsureAllContent (called on save) will fetch all layers for all existing variants (eg: platforms) of the image.

We could do better and only fetch the (possibly missing) layers for the platform being saved.

[apostasie]

@yankay quick and dirty tentative patch: #3920

If the CI passes, maybe give it a spin and check it does fix the problem in your context?

Thanks.

[yankay]

@yankay quick and dirty tentative patch: #3920

If the CI passes, maybe give it a spin and check it does fix the problem in your context?

Thanks.

OK, I'll try it after the CI pass :-)