Architecture-independent (*_all.deb) packages get indexed yet are invisible for apt

[unix-junkie]

I attempted to build and publish a couple of architecture-independent packages (autotools-dev, debhelper).

I can see the package meta-information properly indexed at http://localhost/app/debrepo/debian/dists/jessie/main/binary-all/Packages. Also, noarch binary packages are available under http://localhost/app/debrepo/debian/pool/main/.

At the same time, apt and and its front-ends (aptitude) don't see any of the architecture-independent packages, while architecture-specific ones (i386, amd64, x32) are perfectly visible. I don't experience anything similar with ftp.debian.org or any of the standard mirrors. I believe additional push-ups are required for architecture-independent packages.

I'm running 3 boxes under Debian Jessie.

[netwolfuk]

Thanks for your bug report. I remember reading about these packages and the structure in the repository looks very similar to the other architecture specific ones, so I presumed it would work the same way.

However, I don't have any examples in my test suite so I obviously didn't test before the competition deadline. I'm not sure I'll get a chance over the weekend to look at it, but will try to get a fix out next week.

[netwolfuk]

I have been able to reproduce the problem you are seeing.
I suspect it is because the server does not request an "all" Package file.
Here is the output from teamcity-server.log when the apt-get update request comes in.

 DebDownloadController:: Returning 404 : All regex tried: /debrepo/tcPlugins/dists/jessie/InRelease 
 DebDownloadController:: Returning 404 : All regex tried: /debrepo/tcPlugins/dists/jessie/Release.gpg 
 DebDownloadController:: Returning 404 : All regex tried: /debrepo/tcPlugins/dists/jessie/Release 
 DebDownloadController:: Returning 404 : All regex tried: /debrepo/tcPlugins/dists/jessie/main/binary-amd64/Packages.bz2 
 DebDownloadController:: Returning 404 : All regex tried: /debrepo/tcPlugins/dists/jessie/main/binary-amd64/Packages.xz 
 DebDownloadController:: Returning 404 : All regex tried: /debrepo/tcPlugins/dists/jessie/main/binary-amd64/Packages.lzma 
 DebDownloadController:: Returning 200 : Packages.gz file exists with the name: /debrepo/tcPlugins/dists/jessie/main/binary-amd64/Packages.gz 

I'm guessing, but since it requests the Release file - which tcDebRepository does not support yet (see issue #3) and therefore returns a 404 - perhaps that file should contain the reference to (and hashes of) the binary-all/Packages.gz file like this one does.

[unix-junkie]

@netwolfuk Thanks for your feedback. I'm attaching the list of URLs my server reports HTTP 404 for, just in case you find it useful.
404.txt

[netwolfuk]

@unix-junkie I have found a more likely cause than my previous hypothesis.

This explains how to work around the case where all packages are not included in arch specific listings.

Architectures
Whitespace separated unique single words identifying Debian machine architectures as described in Architecture specification strings, Section 11.1. The field identifies which architectures are supported by this repository. A client should give a warning if it is has a repository configured which doesn't support the machine architecture(s) configured in the client. Servers are allowed to declare support for an architecture even if they currently don't distribute indexes for this architecture. Clients should treat a missing entry for an architecture-specific index of a supported architecture as if that index file would exist, but is empty. If a server is not specifying the supported architectures with this field client behavior is unspecified in this case.

The presents of the architecture all in this field indicates that the architecture-specific indexes do not include information about Architecture:all packages and have instead their own index file with the architecture all. Clients must download the all index files in this case, but must not download them if the Architectures field does not include all.

My reading of that implies that arch specific Packages listings (eg, Packages.gz) should also contain packages for the all architecture.

I'll work on that first, as it's a much quicker fix than implementing the full Releases file support, although I still will need to do that for issue #3. However, issue #3 will be a much simpler problem to solve once issue #4 is implemented (which I also made some progress on last week).

[netwolfuk]

@unix-junkie I have a working solution ready for testing. There is currently no UI, but it's possible to edit the config file and it will be reloaded by the plugin.

Commit c7e0263 adds the ability to configure which architectures are represented by the "all" architecture. When a package with an arch of all, is found in the repository, it is added to the Packages files for all architectures specified in the repository configuration.

The easiest way to enable this is by doing the following:

1. Create a new temporary repository. This will create the section required in the config file.
2. Wait a few seconds and then edit .BuildServer/config/deb-repositories.xml
3. Copy the elements for <architectures-represented-by-all> to your existing repository configuration in that same file and save it. You can remove or add items in that list as appropriate.
4. Wait a few seconds for the file to reload and then navigate to http://localhost/app/debrepo/debian/dists/jessie/main/ and you should see all your configured architectures represented.
5. Navigate inside, and the Packages file should contain your all package(s).

Note: You shouldn't need to run another build as the Package files are generated on the fly (currently, although this will probably change in 1.1).
Those architectures should not show up in the repository browser unless there is at least one "all" package indexed in that repository for that release (jessie) and component (main).

An example <architectures-represented-by-all> element looks like this:

    <architectures-represented-by-all>
        <arch>amd64</arch>
        <arch>arm64</arch>
        <arch>armel</arch>
        <arch>armhf</arch>
        <arch>i386</arch>
        <arch>mips</arch>
        <arch>mipsel</arch>
        <arch>powerpc</arch>
        <arch>ppc64el</arch>
        <arch>s390x</arch>
    </architectures-represented-by-all>

I'll add a bunch of checkboxes to the edit repository UI and then release 1.0.1, but if you want to test it in the mean time, you can download a build from teamcity.jetbrains.com

[unix-junkie]

@netwolfuk Thanks! I'll pull, test and get back to you shortly.

[netwolfuk]

@unix-junkie I probably won't get the UI changes done for another couple of days, so please do test this change before waiting for the UI. Thanks!

[netwolfuk]

@unix-junkie I was shown your article by @grundic (using google translate) and you raise a couple of good points.

1. I will add a way to copy a filter to ease creation of repetitive filters for different builds.
2. I already plan to implement a way to request a re-index of existing builds when new filter rules are added.

Over and above the issues raised on Github, I have a ToDo list here. I will add these two items to it.

[unix-junkie]

@netwolfuk You were right about arch-specific Packages* listings -- I've examined the HTTP traffic between the clients (apt processes) and existing Debian repositories, and indeed, packages for the all architecture are just there. I'm attaching a sample *.pcap file, just in case: apt-get-update.pcap.zip

Speaking of your fix (c7e0263), I didn't even have to edit my ${TEAMCITY_DATA_PATH}/config/deb-repositories.xml -- I just replaced the plug-in ZIP archive with the latest version (82fe341), re-started TeamCity, and my arch-specific Packages* listings started showing architecture-independent packages, too. Is this the intended behaviour?

I'm glad you've taken a look at the article. I believe it is being translated into English, so you won't need to use Google Translate any more (to be published in the TeamCity Blog).

[netwolfuk]

@unix-junkie That's good to know that the upgrade went smoothly. It was not the behaviour I observed, so I will write a unit test to validate that.

I suspect it has something to do with the way jaxb is deserialising the XML to the jaxb annotated data class. I went through a few iterations of that logic so maybe my ${TEAMCITY_DATA_PATH}/config/deb-repositories.xml file was not exactly the same as yours. Thanks for the update.

Good to know about the article. I should hurry up and release a new version of the plugin with architecture independent package support.