First and last images are not detected when loading images to the cluster

[6uliver]

Expected behavior

The second time I start the project the image should be "Found" instantly since the Kubernetes node has the image.

If I would run kubectl get nodes -ojsonpath='{@.items[*].status.images[*].names[*]}' | tr " " "\n" I would get the following:

docker.io/library/service:e1aab03877e36438a59d1832b4e1e69dc3f990527e3e37e177cbc91a88a5b653
docker.io/rancher/klipper-helm@sha256:87db3ad354905e6d31e420476467aefcd8f37d071a8f1c8a904f4743162ae546
docker.io/rancher/klipper-helm:v0.8.3-build20240228
docker.io/rancher/mirrored-library-traefik@sha256:ca9c8fbe001070c546a75184e3fd7f08c3e47dfc1e89bff6fe2edd302accfaec
docker.io/rancher/mirrored-library-traefik:2.10.5
docker.io/rancher/mirrored-metrics-server@sha256:20b8b36f8cac9e25aa2a0ff35147b13643bfec603e7e7480886632330a3bbc59
docker.io/rancher/mirrored-metrics-server:v0.7.0
docker.io/rancher/local-path-provisioner@sha256:aee53cadc62bd023911e7f077877d047c5b3c269f9bba25724d558654f43cea0
docker.io/rancher/local-path-provisioner:v0.0.26
docker.io/rancher/mirrored-coredns-coredns@sha256:a11fafae1f8037cbbd66c5afa40ba2423936b72b4fd50a7034a7e8b955163594
docker.io/rancher/mirrored-coredns-coredns:1.10.1
docker.io/rancher/klipper-lb@sha256:558dcf96bf0800d9977ef46dca18411752618cd9dd06daeb99460c0a301d0a60
docker.io/rancher/klipper-lb:v0.4.7
docker.io/rancher/mirrored-pause@sha256:74c4244427b7312c5b901fe0f67cbc53683d06f4f24c6faee65d4182bf0fa893
docker.io/rancher/mirrored-pause:3.6

And the skaffold dev command's output should contain this and it should continue instantly:

Loading images into k3d cluster nodes...
 - service:e1aab03877e36438a59d1832b4e1e69dc3f990527e3e37e177cbc91a88a5b653 -> Found

Actual behavior

The second time I start the project the image is not "Found" but will be "Loaded" again despite the fact that the Kubernetes node has the image.

If I run kubectl get nodes -ojsonpath='{@.items[*].status.images[*].names[*]}' | tr " " "\n" I get the following:

docker.io/library/service:e1aab03877e36438a59d1832b4e1e69dc3f990527e3e37e177cbc91a88a5b653
docker.io/rancher/klipper-helm@sha256:87db3ad354905e6d31e420476467aefcd8f37d071a8f1c8a904f4743162ae546
docker.io/rancher/klipper-helm:v0.8.3-build20240228
docker.io/rancher/mirrored-library-traefik@sha256:ca9c8fbe001070c546a75184e3fd7f08c3e47dfc1e89bff6fe2edd302accfaec
docker.io/rancher/mirrored-library-traefik:2.10.5
docker.io/rancher/mirrored-metrics-server@sha256:20b8b36f8cac9e25aa2a0ff35147b13643bfec603e7e7480886632330a3bbc59
docker.io/rancher/mirrored-metrics-server:v0.7.0
docker.io/rancher/local-path-provisioner@sha256:aee53cadc62bd023911e7f077877d047c5b3c269f9bba25724d558654f43cea0
docker.io/rancher/local-path-provisioner:v0.0.26
docker.io/rancher/mirrored-coredns-coredns@sha256:a11fafae1f8037cbbd66c5afa40ba2423936b72b4fd50a7034a7e8b955163594
docker.io/rancher/mirrored-coredns-coredns:1.10.1
docker.io/rancher/klipper-lb@sha256:558dcf96bf0800d9977ef46dca18411752618cd9dd06daeb99460c0a301d0a60
docker.io/rancher/klipper-lb:v0.4.7
docker.io/rancher/mirrored-pause@sha256:74c4244427b7312c5b901fe0f67cbc53683d06f4f24c6faee65d4182bf0fa893
docker.io/rancher/mirrored-pause:3.6

The skaffold dev command's output contains this and it's loading for a couple of seconds/minutes at this point depending on the image size:

Loading images into k3d cluster nodes...
 - service:e1aab03877e36438a59d1832b4e1e69dc3f990527e3e37e177cbc91a88a5b653 -> Loaded

Information

I debugged Skaffold source on my local machine and I discovered the problem, Skaffold can not detect the first and last images because of an escaping problem. Because of this it's hard to reproduce, you need to have luck to have the problematic image at the first or at the last place. I'm going to send a PR soon.

• Skaffold version: v2.12.0
• Operating system: MacOS (haven't tested on other OSs but this must be a problem on them)
• Installed via: Homebrew, validated by a local build from source
• Contents of skaffold.yaml:

apiVersion: skaffold/v3
kind: Config
metadata:
  name: my-project
build:
  tagPolicy:
    gitCommit:
      variant: CommitSha
  artifacts:
    - image: service
      context: service
deploy:
  helm:
    releases:
      - name: my-project
        chartPath: chart
        setValueTemplates:
          image: "{{.IMAGE_FULLY_QUALIFIED_service}}"

Steps to reproduce the behavior

1. I start my project with skaffold dev and everything is going to be built and running
2. I stop skaffold with Ctrl+C
3. I start my project again with skaffold dev and everything is going to be built and running

[6uliver]

@ericzzzzzzz Thank you very much! Do you know when will be the next release which will contain this fix?

[ComaVN]

This is the exact same PR I created almost 2 months ago: #9408 #9423

While I'm grateful this bugfix is now merged, for future reference, I'd like to know what I should have done differently to get attention for that PR & issue.

[6uliver]

This is the exact same PR I created almost 2 months ago: #9408 #9423

While I'm grateful this bugfix is now merged, for future reference, I'd like to know what I should have done differently to get attention for that PR & issue.

Sorry, I haven't seen yours otherwise I would have commented on that one, I came up with the same solution on my own.
I agree with your comment, it's great that the fix is merged and I'm also curious why your PR couldn't be merged.

@ericzzzzzzz could you give feedback for us about the issues and PRs, what was wrong with the previous one?

[ericzzzzzzz]

@ComaVN sorry, it was my bad. I forgot that we already had your pr.

[ComaVN]

@ComaVN sorry, it was my bad. I forgot that we already had your pr.

No worries, just glad it's fixed 👍