pkgin on Mac upgrade fails at perl-5.30.2

[mmayer]

$ defaults read "/System/Library/CoreServices/SystemVersion" ProductVersion
10.15.4

$ sudo pkgin -y update && sudo pkgin -y upgrade
processing remote summary (https://pkgsrc.joyent.com/packages/Darwin/trunk/x86_64/All)...
database for https://pkgsrc.joyent.com/packages/Darwin/trunk/x86_64/All is up-to-date
calculating dependencies...done.

1 package to upgrade:
perl-5.30.2

0 to refresh, 1 to upgrade, 0 to install
0B to download, -2.6M to install

upgrading perl-5.30.2...
pkg_install warnings: 0, errors: 1
pkg_install error log can be found in /var/db/pkgin/pkg_install-err.log

$ tail /var/db/pkgin/pkg_install-err.log
pkg_add: 1 package addition failed
---Apr 14 22:23:44: upgrading perl-5.30.2...
pkg_add: Can't open +CONTENTS of depending package p5-Net-SSLeay-1.85nb1
pkg_add: 1 package addition failed
---Apr 14 22:24:43: upgrading perl-5.30.2...
pkg_add: Can't open +CONTENTS of depending package p5-Net-SSLeay-1.85nb1
pkg_add: 1 package addition failed
---Apr 14 22:27:42: upgrading perl-5.30.2...
pkg_add: Can't open +CONTENTS of depending package p5-Net-SSLeay-1.85nb1
pkg_add: 1 package addition failed

As far as I can tell, p5-Net-SSLeay-1.85nb1 no longer even exists as binary package.

$ pkgin avail | grep SSLeay
p5-Crypt-SSLeay-0.72nb7 Crypt::SSLeay - OpenSSL glue that provides LWP https support
p5-Net-SSLeay-1.88 Perl5 module for using OpenSSL

[jperkin]

Yeh, this is a known issue that I'm having trouble reproducing in a debug environment. There's a problem where the list of +REQUIRED_BY packages gets out of sync and not updated correctly during an upgrade, and then fails for subsequent operations, but even with a test environment that performs random upgrades in a loop I can't seem to trigger it, though I've seen it myself locally in the past, just not at a time when I can debug it.

I'll try to get back to my test suite to see if I can find the correct way to trigger it.

[mmayer]

Thanks for your prompt reply. Would it help if I provided my DB file or any other related files?
And can I recover from this condition in some way?
Thanks!

[jperkin]

You can recover by manually editing the broken +REQUIRED_BY files, so in your case /opt/pkg/.pkgdb/perl-5.30.2/+REQUIRED_BY - you will likely see that there are two lines for p5-Net-SSLeay, one of which will be the broken one and one the version you have installed, so removing the broken one should get you past this.

It's interesting that this has been seen before with this particular package combination, there might be something to go on.

Unfortunately sometimes this bug can be quite widespread, I've seen it happen across a number of ruby packages, and that was quite a pain to go through and manually remove all the broken entries.

[jperkin]

Hmm, I might have found something, if you look in your /var/db/pkgin/pkg_install-err.log file and search for the perl-5.30.1 upgrade (or similar), are there a bunch of warnings afterwards, like:

pkg_add: Dependency of p5-Net-SSLeay-1.85nb1 fulfilled by perl-5.28.2, but not by perl-5.30.1
pkg_add: Dependency of p5-Socket6-0.29 fulfilled by perl-5.28.2, but not by perl-5.30.1
pkg_add: Dependency of p5-Net-IP-1.26nb6 fulfilled by perl-5.28.2, but not by perl-5.30.1

?

[jperkin]

Ok yep, that's the bug, I've finally been able to reproduce this in my test suite:

$ ./pkgtest.sh 
Installing agrin-1.0
Installing bepretty-1.0
Installing cohelper-1.0
Installing duckweed-1.0
Installing excitement-1.0
Installing feminate-1.0
Installing gaspingly-1.0
Installing hassock-1.0
Installing imputrescence-1.0
pkg_add: Dependency of imputrescence-1.0 fulfilled by duckweed-1.0, but not by duckweed-2.0
Upgrade 1, pkgs 9 -> 9 -> 9
pkg_add: Dependency of cohelper-1.0nb1 fulfilled by agrin-1.0nb1, but not by agrin-2.0
pkg_add: Dependency of feminate-2.0 fulfilled by agrin-1.0nb1, but not by agrin-2.0
pkg_add: Dependency of feminate-2.0 fulfilled by bepretty-1.0nb1, but not by bepretty-2.0
Upgrade 2, pkgs 9 -> 9 -> 9
pkg_add: Can't open +CONTENTS of depending package cohelper-1.0nb1
pkg_add: 1 package addition failed

The problem occurs when there is a strict dependency match of pkg>=x<y and then an upgrade happens for >=y. I'll start taking a look at a fix.

[jperkin]

Proposed patch:

diff --git a/pkgtools/pkg_install/files/delete/pkg_delete.c b/pkgtools/pkg_install/files/delete/pkg_delete.c
index e9c73577cbb4..87656e87edbd 100644
--- a/pkgtools/pkg_install/files/delete/pkg_delete.c
+++ b/pkgtools/pkg_install/files/delete/pkg_delete.c
@@ -597,17 +597,28 @@ remove_pkg(const char *pkg)
         * Errors in the remaining part are counted, but don't stop the
         * processing.
         */
-
        for (p = plist.head; p; p = p->next) {
-           if (p->type != PLIST_PKGDEP)
-               continue;
-           if (Verbose)
-               printf("Attempting to remove dependency "
-                      "on package `%s'\n", p->name);
-           if (Fake)
-               continue;
-           match_installed_pkgs(p->name, remove_depend,
+               char *depmatch, *sep;
+               if (p->type != PLIST_PKGDEP)
+                       continue;
+               if (Verbose)
+                       printf("Attempting to remove dependency "
+                           "on package `%s'\n", p->name);
+               if (Fake)
+                       continue;
+               /*
+                * Convert any specific version match into a general match
+                * to catch any cases where a package has been forcibly
+                * upgraded to a version that no longer matches.
+                */
+               depmatch = xstrdup(p->name);
+               if ((sep = strpbrk(depmatch, "<>")) != NULL) {
+                       *sep = '\0';
+                       depmatch = xasprintf("%s-[0-9]*", depmatch);
+               }
+               match_installed_pkgs(depmatch, remove_depend,
                                 __UNCONST(pkg));
+               free(depmatch);
        }
 
        free_plist(&plist);

I've been running this in my test script for a while and it's working as expected. Further testing required.

[mmayer]

Thanks for your help.

Unfortunately, there was one mismatch after the next once I started editing +REQUIRED_BY (i.e. it would complain about the subsequent package and then the next). So, I ended up recursively uninstalling Perl and then reinstalling it and the packages I had before. Luckily, it wasn't too painful.

As for the multiple entries in my pkg_install-err.log, I was just running update & upgrade a few times in a row to see if it would sort itself out after another update.

[jperkin]

I've pushed a fixed version of pkg_install out to the trunk release for wider testing, though this will only help once installed.

If the pkgdb becomes out of sync again when upgrading to the fixed version, an easier fix is to run pkg_admin rebuild && pkg_admin rebuild-tree, sorry that I wasn't aware of the latter previously (not sure how I missed it all this time!)

[jperkin]

This was fixed with recent changes in pkg_install and pkgin.