Add manifestcache push for tag and digest to local repository

[raphaelzoellner]

Comprehensive Summary of your change

Harbor Proxies will push a manifest referencing a digest and tag to the local repo if both are known.

Issue being fixed

Fixes #21122

Please indicate you've done the following:

• Well Written Title and Summary of the PR
• Label the PR as needed. "release-note/ignore-for-release, release-note/new-feature, release-note/update, release-note/enhancement, release-note/community, release-note/breaking-change, release-note/docs, release-note/infra, release-note/deprecation"
• Accepted the DCO. Commits without the DCO will delay acceptance.
• Made sure tests are passing and test coverage is added if needed.
• Considered the docs impact and opened a new docs issue or PR with docs changes if needed in website repository.

[stonezdj]

Thanks for your contribution! because the current code change doesn't fix the issue #21122, and might result in regression issue, so we will close this PR.

[raphaelzoellner]

Hello @stonezdj,

could you give a hint why this code change (creating the manifest referenced by the tag) wouldn't solve or at least improve the described behavior in #21122, so one can think about another solution?

Is this maybe a misunderstanding with the PR #21123, which is already closed?

[stonezdj]

You have replace the m.local.PushManifest(art.Repository, getReference(art), man) with the m.push(art, man),
the intention of this method is push the manifest to the proxy cache project if there is any tag exist in the art, current the art is parsing from the pull request, for a normal image pull request, it is pulled either by digest or by tag. it can't be both.
The previous getReference(art) will return tag if it pull by tag, and return digest if it is pull by digest.
Current code change has no difference with the previous implementation.

[raphaelzoellner]

You have replace the m.local.PushManifest(art.Repository, getReference(art), man) with the m.push(art, man)
Yes my intention is to push the manifest to the local proxy cache project with any reference known (tag, digest or both).

for a normal image pull request, it is pulled either by digest or by tag. it can't be both.
I agree the GET manifest request either uses a tag or digest as reference, but during the storage of the proxied manifest the digest might be looked up in the remote repository and the manifest is then only stored under the digest in the local proxy cache project.

The previous getReference(art) will return tag if it pull by tag, and return digest if it is pull by digest.
I think there are cases where art will contain both a digest and tag here. In this case the getReference(art) will favor digest over tag, ultimately leading to the manifest being stored in the local proxy cache project only under the digest.

This happens during the proxy controller's ProxyManifest call.

For a manifest pull referencing a tag the digest will be looked up in the remote repository.
https://github.com/goharbor/harbor/blob/v2.10.2/src/controller/proxy/controller.go#L224

Then the digest will be added to the copied artInfo which should until then contain the tag parsed from the the original pull by tag.
https://github.com/goharbor/harbor/blob/v2.10.2/src/controller/proxy/controller.go#L249

This artInfo containing both tag and digest will then be used in the original call to m.local.PushManifest(art.Repository, getReference(art), man), where getReference(art) leads to the digest being favored over the tag.
https://github.com/goharbor/harbor/blob/v2.10.2/src/controller/proxy/controller.go#L251C1-L251C69
https://github.com/goharbor/harbor/blob/v2.10.2/src/controller/proxy/controller.go#L318
https://github.com/goharbor/harbor/blob/v2.10.2/src/controller/proxy/manifestcache.go#L201
https://github.com/goharbor/harbor/blob/v2.10.2/src/controller/proxy/controller.go#L327-L330

My proposal is to push the manifest with both references (digest and tag) if both are known. This should lead to subsequent manifest pulls referencing the tag being served from the local proxy cache project if the digest has not changed to the remote manifest digest for said tag.

[raphaelzoellner]

Hello @stonezdj,

are there any concerns left, or could I provide anything else?

[nasseredine]

Hello @raphaelzoellner,
Can you please confirm for my own understanding that for subsequent artefact pulls referencing a tag, there is still a Manifest HEAD request to check the docker-content-digest or etag?

[raphaelzoellner]

Hello @nasseredine ,

I think this should be achieved by the call to proxy controller's UseLocalManifest method in the repoproxy middleware here.
https://github.com/goharbor/harbor/blob/v2.13.0/src/server/middleware/repoproxy/proxy.go#L201

During a manifest get by tag request, art should contain only the tag at this point.
For a subsequent request, the artifact should already exist in the local proxy cache.
https://github.com/goharbor/harbor/blob/v2.13.0/src/controller/proxy/controller.go#L157

The existence of the manifest will then be checked on the remote registry via the clients ManifestExist method, that performs a HEAD request to lookup the digest among other things.
https://github.com/goharbor/harbor/blob/v2.13.0/src/controller/proxy/controller.go#L167
https://github.com/goharbor/harbor/blob/main/src/controller/proxy/remote.go#L105
https://github.com/goharbor/harbor/blob/v2.13.0/src/pkg/registry/client.go#L266

For manifests, the digest of the remote manifest is then compared to the digest of the artifact in the local cache to decide whether to serve from the local cache (digest matches) or not (digest doesn't match).
https://github.com/goharbor/harbor/blob/v2.13.0/src/controller/proxy/controller.go#L181
https://github.com/goharbor/harbor/blob/v2.13.0/src/controller/proxy/controller.go#L194

For manifest lists the digest is not compared and the manifest list is served without digest comparison.
https://github.com/goharbor/harbor/blob/v2.13.0/src/controller/proxy/controller.go#L202

But since the proposed changes only update the behavior of ManifestCache CacheContent
the https://github.com/goharbor/harbor/blob/v2.13.0/src/controller/proxy/manifestcache.go#L177

and are not affecting ManifestListCache CacheContent
https://github.com/goharbor/harbor/blob/v2.13.0/src/controller/proxy/manifestcache.go#L66

I would assume the behavior for manifest list caches to remain the same.

[nasseredine]

Thank you very much for your detailed explanation, @raphaelzoellner. The provided code references helped me understand the handling logic of proxy cache manifests.

[Vad1mo]

Approve, that we need that functionality. I didn't try it out on the technical side yet.

[raphaelzoellner]

Hi, is there any update on this?

[llCorvinSll]

Hi, It seems we are facing the same issue. (manifests are not cached and while upstream is not available, fetching fails)

[raphaelzoellner]

Hi,
this PR has been opened for quite some time. Are there any concerns left or any additional steps necessary?

The proposed changes are blocked by 1 missing approval. Could one more Harbor maintainer review this?
@wy65701436
@AllForNothing
@MinerYang

[chris93111]

Hi, It seems we are facing the same issue, there any update on this?

[Copilot]

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 46.51%. Comparing base (c8c11b4) to head (78ae0f8).
Report is 470 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #21141      +/-   ##
==========================================
+ Coverage   45.36%   46.51%   +1.14%     
==========================================
  Files         244      253       +9     
  Lines       13333    14236     +903     
  Branches     2719     2925     +206     
==========================================
+ Hits         6049     6622     +573     
- Misses       6983     7264     +281     
- Partials      301      350      +49     

Flag	Coverage Δ
unittests	46.51% <ø> (+1.14%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 178 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[stonezdj]

@raphaelzoellner please rebase the code with main branch to merge the code

[raphaelzoellner]

I think the github ci/cd workflow errors of the last run seem to be unrelated to the proposed changes.
I've rebased to see if the APITEST errors persist.