Cannot publish repo where two different versions of a package were added using the same filename

[reynir]

Detailed Description

When adding two packages with differing versions and content from files both named builder-web.deb to a repository and then subsequently publishing the repo I don't expect an error, but I observe the following error:

Loading packages...
Generating metadata files and linking package files...
ERROR: unable to publish: unable to process packages: error linking file to /home/reynir/.aptly/public/pool/main/b/builder-web/builder-web.deb: file already exists and is different

The output of aptly package show builder-web is:

ackage: builder-web
Priority: optional
Section: unknown
Maintainer: Robur Team <team@robur.coop>
Architecture: all
Version: 0.1.0-89-g338fa9d
Depends: libgmp10, libsqlite3-0, libev4
Filename: builder-web.deb
Size: 11472404
MD5sum: e61a8912d118a6d9ac84a09cd2da4be1
SHA1: 852a46863f591ca01a2d7164177e1f03133b9018
SHA256: b995cd80d28abd86bc1c3347b1a0e15aef63ded372a7d8547351319008aa4e7c
SHA512: 45c2c757c745eeb3df85a7c15527ad08dbc9dbe78bd9c19e670d5b46b865ffc7c208f43ee16687a29c946f5ee756864236e88b1e7fd2905b252d816537a62bbc
Description: Web service for storing and presenting builds.
  Builder-web stores builds in a sqlite database and serves them via HTTP.
Homepage: https://git.robur.io/robur/builder-web
Standards-Version: 4.4.1
Vcs-Browser: https://git.robur.io/robur/builder-web
Vcs-Git: https://git.robur.io/robur/builder-web.git

Package: builder-web
Priority: optional
Section: unknown
Maintainer: Robur Team <team@robur.coop>
Architecture: all
Version: 0.1.0-88-g9333773
Depends: libgmp10, libsqlite3-0, libev4
Filename: builder-web.deb
Size: 11470588
MD5sum: eb9fcfc2f82eb176a01ce5b04363d6b4
SHA1: 5d2a696ded8876ff0734f0b156290597aadd7600
SHA256: 5445eeb759d1b07138c31ead93d253ab56e0ae577753befa053232ea2cdedcd2
SHA512: a98f8502103e612d54606e814547ca37d2190c2e9ba526026ae6ea0770f0a20aa6e862a7a3056d7aa973af943a77e2b1276fc309b04c05287442eacab76e2d66
Description: Web service for storing and presenting builds.
  Builder-web stores builds in a sqlite database and serves them via HTTP.
Homepage: https://git.robur.io/robur/builder-web
Standards-Version: 4.4.1
Vcs-Browser: https://git.robur.io/robur/builder-web
Vcs-Git: https://git.robur.io/robur/builder-web.git

As the output says the versions differ (0.1.0-89-g338fa9d vs 0.1.0-88-g9333773) so it should be fine according to the documentation.

Issue #439 may be related (the reporter says they add a random prefix to the filename).

Context

Our packaging setup creates .deb files using the same name every time. Other users may benefit from not having to worry about name clashes in filenames.

Possible Implementation

Perhaps a unique name could be generated, or the behavior could be documented -- I was not able to find this behavior documented.

Your Environment

Aptly version 1.4.0+ds1-4+b4 from the debian repository

[Vitexus]

I have similar issue:

Generating metadata files and linking package files...
ERROR: unable to publish: unable to process packages: link /var/lib/jenkins/workspace/RebuildDebRepo/pool/04/90/d5360e42e81ffbd1f4d38703ceda_php-vitexsoftware-ease-core_1.37~buster~56_all.deb /var/www/html/repo.vitexsoftware.cz/pool/main/p/php-vitexsoftware-ease-core/php-vitexsoftware-ease-core_1.37~buster~56_all.deb: no such file or directory
Loading packages...
Generating metadata files and linking package files...
ERROR: unable to publish: unable to process packages: link /var/lib/jenkins/workspace/RebuildDebRepo/pool/44/58/4b4a2b8f5b870f66226410a126f6_php-vitexsoftware-ease-core_1.37~bullseye~56_all.deb /var/www/html/repo.vitexsoftware.cz/pool/main/p/php-vitexsoftware-ease-core/php-vitexsoftware-ease-core_1.37~bullseye~56_all.deb: no such file or directory
Loading packages...
Generating metadata files and linking package files...
ERROR: unable to publish: unable to process packages: link /var/lib/jenkins/workspace/RebuildDebRepo/pool/33/a5/0033615744e0d8c0cafaeac67e28_composer_2.2.6~focal~28_all.deb /var/www/html/repo.vitexsoftware.cz/pool/backports/c/composer/composer_2.2.6~focal~28_all.deb: no such file or directory
Loading packages...
Generating metadata files and linking package files...
ERROR: unable to publish: unable to process packages: link /var/lib/jenkins/workspace/RebuildDebRepo/pool/ec/70/6206d454faf9b6ff494a8b44086a_composer_2.2.6~hirsute~28_all.deb /var/www/html/repo.vitexsoftware.cz/pool/backports/c/composer/composer_2.2.6~hirsute~28_all.deb: no such file or directory
Loading packages...
Generating metadata files and linking package files...
ERROR: unable to publish: unable to process packages: link /var/lib/jenkins/workspace/RebuildDebRepo/pool/cd/85/fe1aa769c3b5ac30b4df83611310_composer_2.2.6~impish~28_all.deb /var/www/html/repo.vitexsoftware.cz/pool/backports/c/composer/composer_2.2.6~impish~28_all.deb: no such file or directory
Loading packages...
Generating metadata files and linking package files...
ERROR: unable to publish: unable to process packages: link /var/lib/jenkins/workspace/RebuildDebRepo/pool/73/53/59dac9c6702d43ab18427072ed1a_php-vitexsoftware-ease-core_1.37~bookworm~56_all.deb /var/www/html/repo.vitexsoftware.cz/pool/main/p/php-vitexsoftware-ease-core/php-vitexsoftware-ease-core_1.37~bookworm~56_all.deb: no such file or directory

[reynir]

Hi @Vitexus. It seems from the error messages that our issues are different. Mine is about a file existing but differs from what aptly expects. Yours is about a file that seems to be missing entirely.

[neolynx]

Hi @reynir,

Debian package files usually follow a naming format: $name_$version_$architecture.deb

Aptly seems to expect the files in this format, and does not do any checks or renaming for internal storage/reference.

Is it possible for you to follow that naming convention ? Otherwise we could open a feature request for internally naming deb file uniquely...

Kind Regards,

André

[neolynx]

Hi @Vitexus,

this seems to be a different issue, maybe you could open a new one, explaining what steps lead to the state where these files cannot be found ?

Kind Regards,

André

[Vitexus]

this seems to be a different issue, maybe you could open a new one, explaining what steps lead to the state where these files cannot be found ?

As you say. But now i have not much evidence to start new issue. I must investigate it a lot to figure out when the problem happens.

[reynir]

Hi André / @neolynx,

We are now following that naming scheme. I think this should be documented or changed. Smira says in the above linked issue

Package filename shouldn't matter that much (or shouldn't matter at all).

[r4co0n]

It should be relatively trivial to always use <Package>_<Version>_<Architecture>.<Package-Type> as filename in the publication's pool, as specified by the imported package's DEBIAN/control file - Relevant fields are defined in Debian Policy Manual Chapter 5.

If I am not mistaken, the filename of the package to be imported could always be this_would_never_work.funky_pkg, as long as the file represents an installable Debian package and DEBIAN/control does not claim to match any package currently in the publication's pool with the same:

• Package
• Version
• Architecture
• Package-Type

control fields while the hashes of both files are different,

Irrelevant concerns based on an imagined central pool sorting by more than a long hash:

• If the hashes (and the control fields) match, the call would be a superfluous no-op.

[devopstt]

Same for Docker Repo :/
https://download.docker.com/linux/ubuntu/
ERROR: unable to publish: unable to process packages: error linking file to /opt/mirror/public/3rdparty/2022-03-15/pool/main/c/containerd.io/containerd.io_1.5.10-1_amd64.deb: file already exists and is different
The files are indeed different but have the same name for bionic and focal.
https://download.docker.com/linux/ubuntu/dists/focal/pool/stable/amd64/containerd.io_1.5.10-1_amd64.deb
https://download.docker.com/linux/ubuntu/dists/bionic/pool/stable/amd64/containerd.io_1.5.10-1_amd64.deb

What can I even do here?

[dhs-rec]

The same that Debian and Ubuntu also do. Add a distribution tag to the version, like "1.5.10-1ubuntu18.04" and "1.5.10-1ubuntu20.04". This is explained in great detail here: https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-version

[devopstt]

I mean, its not like I can change the way Docker publishes their packages :D

[dhs-rec]

You could try...

[jamesps-ebi]

Hi @reynir,

Debian package files usually follow a naming format: $name_$version_$architecture.deb

Aptly seems to expect the files in this format, and does not do any checks or renaming for internal storage/reference.

Is it possible for you to follow that naming convention ? Otherwise we could open a feature request for internally naming deb file uniquely...

Kind Regards,

André

I believe I have the same issue with my aptly repo which is published to s3.
We're automatically downloading the latest Zoom client package from https://zoom.us/client/latest/zoom_amd64.deb with wget and adding it to the repo nightly.
The problem is although the package version/content may differ, the filename is always the same. When I try to re-publish to s3, I get the error message:

Loading packages...
Generating metadata files and linking package files...
ERROR: unable to publish: unable to process packages: error putting file to ubuntu/pool/main/z/zoom/zoom_amd64.deb: file already exists and is different: S3: us-east-1:example_bucket/ubuntu

I suppose we can work around this by prefixing the output filename with a date/time value and using the -remove-files option when adding the package to the repo.
But it would be useful if aptly can generate a unique name internally using the package contents values like you suggested