Frequent problems with `pkgin upgrade` or `pkgin full-upgrade` and a few packages, especially sqlite3

[drboone]

Per brief IRC discussion:

it's a real issue that, the same as the "pkg conflicts with ", I've not had a situation where I can reproduce and fix it so yeh, please raise an issue and include any information you can, e.g. a tarball of pkgdb would be handy

raising this issue about conflicts of a package with itself in the hope that I have data that might help track down the problem.

Extracts related to sqlite3 from pkg_install-err.log:

---Jan 09 14:41:37: upgrading sqlite3-3.40.0nb1...
---Feb 04 08:14:27: upgrading sqlite3-3.40.1...
pkg_delete: couldn't entirely delete package `sqlite3-3.40.0nb1'
---Mar 01 08:14:32: upgrading sqlite3-3.41.0...
---Mar 12 08:16:32: refreshing sqlite3-3.41.0...
pkg_add: Conflicting PLIST with sqlite3-3.39.0: bin/sqlite3
---Mar 13 08:12:27: upgrading sqlite3-3.41.1...
pkg_add: Conflicting PLIST with sqlite3-3.39.0: bin/sqlite3
---Mar 14 08:09:57: upgrading sqlite3-3.41.1...
pkg_add: Conflicting PLIST with sqlite3-3.39.0: bin/sqlite3
---Mar 15 08:11:58: upgrading sqlite3-3.41.1...
pkg_add: Conflicting PLIST with sqlite3-3.39.0: bin/sqlite3
---Mar 17 08:12:08: upgrading sqlite3-3.41.1...
pkg_add: Conflicting PLIST with sqlite3-3.39.0: bin/sqlite3

pkgdb.byfile.db is attached, gzipped because %^&*( github.

pkgdb.byfile.db.gz

[drboone]

Here's a longer log extract for a machine that's currently exhibiting the sqlite3 issue.

bigriver.txt

[drboone]

I have several machines today that are having trouble upgrading openssl. It's an easy workaround -- pkg_delete -f, pkg_add. Log from one:

---Oct 18 14:33:05: [3/4] upgrading openssl-1.1.1w...
pkg_add: Conflicting PLIST with openssl-1.1.1pnb1: bin/c_rehash
pkg_add: 1 package addition failed

There's further weirdness - some packages claim to install, but another full-upgrade will do it over and over. And I've seen e.g. mozilla-rootcerts listed twice in the upgrade list.

[drboone]

Seeing this today on routine pkgin full-upgrade:

---Mar 05 14:52:07: [1/1] upgrading pkg_install-20240126...
pkg_add: Conflicting PLIST with pkg_install-20210410: man/man1/pkg_add.1.gz
pkg_add: 1 package addition failed

[jperkin]

Ugh, yeh, lemme see if I can carve out time tomorrow to try and nail this down once and for all.

[drboone]

If it'd help to have ssh access to an affected machine, I can arrange.

[drboone]

The corrupted database warning you mentioned in IRC the other day has appeared on several of our systems:

beautiful 5 $ pkg_admin rebuild
pkg_admin: corrupt pkgdb, duplicate PKGBASE entries:
        pkgsrc-gnupg-keys-20190423
        pkgsrc-gnupg-keys-20201014

So what's the proper cleanup process here? I'm pretty sure I've removed specific package version in the past, possibly using pkg_add to get key packages back.

[drboone]

Digging deeper, I'll add that the most recent gz where I've had pkgin full-upgrade problems does not exhibit the corrupt pkgdb errors, but does still have conflicting file problems this morning:

---Apr 04 12:13:29: [1/19] refreshing ncurses-6.4...
---Apr 04 12:13:29: [2/19] refreshing readline-8.2nb2...
---Apr 04 12:13:29: [3/19] refreshing sqlite3-3.45.2...
---Apr 04 12:13:29: [4/19] refreshing xz-5.4.6...
---Apr 04 12:13:30: [5/19] refreshing ncursesw-6.4...
---Apr 04 12:13:30: [6/19] upgrading pkg_install-20240307...
pkg_add: Conflicting PLIST with pkg_install-20211115: man/man1/pkg_add.1.gz
pkg_add: 1 package addition failed
---Apr 04 12:13:30: [7/19] refreshing python311-3.11.8...
---Apr 04 12:13:32: [8/19] refreshing python312-3.12.2...
---Apr 04 12:13:35: [9/19] upgrading pkg_install-20240307...
pkg_add: Conflicting PLIST with pkg_install-20211115: man/man1/pkg_add.1.gz
pkg_add: 1 package addition failed
---Apr 04 12:13:35: [10/19] refreshing libarchive-3.7.2...
---Apr 04 12:13:35: [11/19] refreshing pkgsrc-gnupg-keys-20231210...
pkg_add: Conflicting PLIST with pkgsrc-gnupg-keys-20201014: share/gnupg/pkgsrc-security.gpg
pkg_add: 1 package addition failed
---Apr 04 12:13:35: [12/19] upgrading pkgin-23.8.1nb3...
---Apr 04 12:13:35: [13/19] refreshing pkgin-23.8.1nb3...
---Apr 04 12:13:36: [14/19] refreshing py312-pip-24.0...
---Apr 04 12:13:36: [15/19] upgrading bsdinstall-20160108nb1...
---Apr 04 12:13:36: [16/19] refreshing py312-wheel-0.43.0...
---Apr 04 12:13:36: [17/19] refreshing py312-setuptools-69.2.0...
---Apr 04 12:13:36: [18/19] refreshing py311-pip-24.0...
---Apr 04 12:13:37: [19/19] refreshing py311-wheel-0.43.0...

This is a machine I'm quite convinced has never had an improper tools or bootstrap kit applied -- it got the tools during the new-machine install, and hasn't been messed with.

[drboone]

Another round of this, still with no errors from pkg_admin rebuild:

---May 13 12:14:09: [1/5] upgrading pkg_install-20240307...
pkg_add: Conflicting PLIST with pkg_install-20211115: man/man1/pkg_add.1.gz
pkg_add: 1 package addition failed
---May 13 12:14:10: [2/5] upgrading pkg_install-20240307...
pkg_add: Conflicting PLIST with pkg_install-20211115: man/man1/pkg_add.1.gz
pkg_add: 1 package addition failed
---May 13 12:14:10: [3/5] refreshing pkgsrc-gnupg-keys-20231210...
pkg_add: Conflicting PLIST with pkgsrc-gnupg-keys-20201014: share/gnupg/pkgsrc-security.gpg
pkg_add: 1 package addition failed
---May 13 12:14:10: [4/5] upgrading pkgin-23.8.1nb3...
---May 13 12:14:10: [5/5] upgrading bsdinstall-20160108nb1...
avenueq 6 $ pkg_admin rebuild

Stored 27252 files and 1 explicit directory from 45 packages in /opt/tools/var/db/pkg/pkgdb.byfile.db.
Done.