Cannot create repository version since there are newly added packages with the same name, version, and architecture - Docker CE

[m-bull]

Version

  "versions": [
    {
      "component": "core",
      "version": "3.43.2",
      "package": "pulpcore",
      "domain_compatible": true
    },
    {
      "component": "deb",
      "version": "3.1.1",
      "package": "pulp_deb",
      "domain_compatible": false
    }
]

Describe the bug
Syncing from https://download.docker.com/linux/ubuntu results in:

'Cannot create repository version since there are newly added packages with the same name, version, and architecture, but a different checksum. If the log level is DEBUG, you can find a list of affected packages in the Pulp log.'

Despite settings.py containing

LOGGING = {'dynaconf_merge': True, 'loggers': {'': {'handlers': ['console'], 'level': 'DEBUG'}}}

No list of affected packages is produced in the Pulp log.

I think that the sync failure is being caused by the containerd.io package, for which there is a duplicate content item with the same name, version and architecture for both Focal and Jammy:

pulp deb content list --package containerd.io --version "1.6.27-1"
[
  {
    "pulp_href": "/pulp/api/v3/content/deb/packages/018d0d02-759d-72f6-a8a0-78a344d6b088/",
    "pulp_created": "2024-01-15T12:04:50.640930Z",
    "artifact": "/pulp/api/v3/artifacts/018d0d02-8a30-7912-8d6f-07449576be84/",
    "relative_path": "dists/focal/pool/stable/amd64/containerd.io_1.6.27-1_amd64.deb",
    "md5": null,
    "sha1": "e9b2e8157fb1ca7f246075ada1f75f89748070d5",
    "sha224": "6f8506fbffcbd5291f6d7a73cbfe6afd5972c73a7061a9cc670247e2",
    "sha256": "9b8975ba8419489455c57c128c2e817973952e54bc380fdbbce1a1f93df05911",
    "sha384": "45645531759e475259aaadafca446aefefe07cc250a8997862bafce6bbb336fac3164de55c3dbfdf7e8c6c9dfeb9c3d8",
    "sha512": "61a5b494e0d65a3f99175382636e7c455cf7bab19aafae11d81704873d289970d8bca4a97e1d50d20a026871c14114b2f8cb1bfed33c697f0a2f39c6cb928bd5",
    "package": "containerd.io",
    "source": null,
    "version": "1.6.27-1",
    "architecture": "amd64",
    "section": "devel",
    "priority": "optional",
    "origin": null,
    "tag": null,
    "bugs": null,
    "essential": null,
    "build_essential": null,
    "installed_size": "115640",
    "maintainer": "Containerd team <help@containerd.io>",
    "original_maintainer": null,
    "description": "An open and reliable container runtime",
    "description_md5": null,
    "homepage": "https://containerd.io",
    "built_using": null,
    "auto_built_package": null,
    "multi_arch": null,
    "breaks": null,
    "conflicts": "containerd, runc",
    "depends": "libc6 (>= 2.14), libseccomp2 (>= 2.5.0)",
    "recommends": null,
    "suggests": null,
    "enhances": null,
    "pre_depends": null,
    "provides": "containerd, runc",
    "replaces": "containerd, runc"
  },
  {
    "pulp_href": "/pulp/api/v3/content/deb/packages/018d0d02-7439-72ea-afed-8e6ac1a7178f/",
    "pulp_created": "2024-01-15T12:04:47.595274Z",
    "artifact": "/pulp/api/v3/artifacts/018d0d02-876e-776f-b009-326ce7ca4da3/",
    "relative_path": "dists/jammy/pool/stable/amd64/containerd.io_1.6.27-1_amd64.deb",
    "md5": null,
    "sha1": "2af3f935500c3a49503a1f57388baea38627c22f",
    "sha224": "05d300223c91fc8bba95f4d98680e459715ef7691eceb64b66d66614",
    "sha256": "79f2f0157591174893feb6d8232b66cb3cf1d9e9542f3f42e278f2c583f3a792",
    "sha384": "911d40e7a4d91e8d6df8e56cc47634051666006e3cec48081c82efba52430130f998011eff384610a37d74d33741c588",
    "sha512": "8c2a4d947a705cc0a3110dbbe63b4224bc1f2dd67f8be74bbcc3aee6a59b91d5b8ba95e2ef0d2286b1487b0324eb0b0b1b6bf671998570705bedd4fc886f33de",
    "package": "containerd.io",
    "source": null,
    "version": "1.6.27-1",
    "architecture": "amd64",
    "section": "devel",
    "priority": "optional",
    "origin": null,
    "tag": null,
    "bugs": null,
    "essential": null,
    "build_essential": null,
    "installed_size": "115640",
    "maintainer": "Containerd team <help@containerd.io>",
    "original_maintainer": null,
    "description": "An open and reliable container runtime",
    "description_md5": null,
    "homepage": "https://containerd.io",
    "built_using": null,
    "auto_built_package": null,
    "multi_arch": null,
    "breaks": null,
    "conflicts": "containerd, runc",
    "depends": "libc6 (>= 2.34), libseccomp2 (>= 2.5.0)",
    "recommends": null,
    "suggests": null,
    "enhances": null,
    "pre_depends": null,
    "provides": "containerd, runc",
    "replaces": "containerd, runc"
  }
]

1. What is the correct course of action to fix the sync failure?
2. I think there may be a bug here that causes the list of packages to not be printed to the log.

To Reproduce
Sync Jammy and Focal dists of the docker-ce Ubuntu repository.

Expected behavior
Sync completes!

[quba42]

Ugh very annoying if the logging is not working as designed here!

If I understood correctly your remote is using distributions="jammy focal". Can you test whether syncing just jammy into one repository, and just focal into another works? At the least this will provide some more information, if it works, it might be an initial workaround.

The error you are getting is not intended to be fixed, since it is really an error with the upstream repository, that can result in undefined behavior in APT clients. However, if rigidly enforcing the APT repo spec in these cases, ends up creating more real world usage problems than it solves, then it may be necessary to rethink how pulp_deb should handle them.

Irrespective of the design discussion, we should fix the logging feature.

[m-bull]

Thanks for the update and advice!

More background - this Pulp just got an upgrade and pulled in this ed225b8, which I assume is why we've just starting hitting this issue. Every matched version of containerd.io from docker-ce for Focal and Jammy have different checksums in the upstream Packages file, but this issue has never surfaced until we upgraded, because it was never checked 😂.

I'll try separating out the APT distributions into different repositories.

If it's helpful, this got me the PK of the package so I could go hunt in the database for it:

diff --git a/repository.py b/repository.matt.py
index 699982e..ea2c86f 100644
--- a/repository.py
+++ b/repository.matt.py
@@ -152,16 +152,9 @@ def handle_duplicate_packages(new_version):
         )
         added_unique = package_qs_added.distinct(*repo_key_fields)
         added_checksum_unique = package_qs_added.distinct(*repo_key_fields, "sha256")
-
         if added_unique.count() < added_checksum_unique.count():
             package_qs_added_duplicates = added_checksum_unique.difference(added_unique)
-            if log.isEnabledFor(logging.DEBUG):
-                message = _(
-                    "New repository version contains multiple packages with '{}', but differing "
-                    "checksum!"
-                )
-                for package_fields in package_qs_added_duplicates.values(*repo_key_fields):
-                    log.debug(message.format(package_fields))
+            log.warning("DUPES: {}".format(package_qs_added_duplicates))
 
             message = _(
                 "Cannot create repository version since there are newly added packages with the "

[quba42]

@m-bull Your understanding is correct. The background is that we always had intended to disallow this kind of duplicate package, but the implementation was buggy so in practice we did not disallow such duplicates in most cases. The change you linked to fixed our original implementation to align with our original design intention. This is why you are now running into this error message, when you did not do so in the past.

If is of course entirely possible that our original design intention (which the latest pulp_deb versions are now actually implementing), is to rigid for real world usage. Once I find the time, I will have a detailed look at this docker repo case in order to try and decide if pulp_deb handling should be amended. Strictly speaking I consider this to be a bug with how docker constructs their APT repo. How to handle faultily constructed (but still usable) upstream repos within pulp_deb is not at all trivial.

[m-bull]

@quba42 - thanks again for your help, separating the distributions into different repositories does allow the sync to progress, which is an okay workaround for now.

Feel free to close this issue!

[quba42]

@m-bull I will keep the issue open for now, and if it is just a reminder to re-test and fix the debug logging option.