Update ci.nodejs.org to the latest LTS Jenkins version

[targos]

We are currently on Jenkins 2.346.3.

Here are the guides to update to 2.361.x and 2.375.x:

• https://www.jenkins.io/doc/upgrade-guide/2.361/
• https://www.jenkins.io/doc/upgrade-guide/2.375/

[richardlau]

We are currently on Jenkins 2.346.3.

Here are the guides to update to 2.361.x and 2.375.x:

If I'm remembering correctly we have Jenkins installed via apt. However I copied over an older jenkins.war file after the last upgrade -- I think I renamed the last upgraded war file (probably with a .bak extension, I'm not in a position to be able to log into the server right now to check) which we probably want to put back before running any apt comands.

[richardlau]

My plan is to update that Jenkins server after next week's security releases are done, but before unlocking the CI.

[richardlau]

My plan is to update that Jenkins server after next week's security releases are done, but before unlocking the CI.

This security release has been really draining, it's 11pm at night and I don't really have the energy to deal with anything more in the event something goes wrong with the update so I'm going to postpone it.

[targos]

If I'm remembering correctly we have Jenkins installed via apt.

Correct. apt info jenkins shows 2.375.3 as the latest available version, which is also correct.

Note:

# apt update
Hit:1 http://mirrors.digitalocean.com/ubuntu jammy InRelease
Ign:2 https://pkg.jenkins.io/debian-stable binary/ InRelease
Hit:3 https://pkg.jenkins.io/debian-stable binary/ Release
Hit:4 http://mirrors.digitalocean.com/ubuntu jammy-updates InRelease
Get:5 https://repos.influxdata.com/ubuntu jammy InRelease [7,019 B]
Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease
Err:5 https://repos.influxdata.com/ubuntu jammy InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY D8FF8E1F7DF8B07E
Fetched 7,019 B in 1s (8,099 B/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
152 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: https://pkg.jenkins.io/debian-stable/binary/Release.gpg: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://repos.influxdata.com/ubuntu jammy InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY D8FF8E1F7DF8B07E
W: Failed to fetch https://repos.influxdata.com/ubuntu/dists/jammy/InRelease  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY D8FF8E1F7DF8B07E
W: Some index files failed to download. They have been ignored, or old ones used instead.

I wonder why we have that repo. InfluxDB isn't installed.

[richardlau]

I wonder why we have that repo. InfluxDB isn't installed.

Maybe the client for grafana?

[richardlau]

There's another advisory coming out next week:

https://groups.google.com/g/jenkinsci-advisories/c/RmkqB1wAY14

The Jenkins project plans to publish new Jenkins releases (weekly, LTS 2.375.4, and LTS 2.387.1) on Wednesday, March 8. These updates will contain fixes for security issues present in current versions of Jenkins. The highest severity is "High". The security advisory will be issued at the same time to provide further information.

I'll aim for Thursday 9 March to re-attempt the update.

[richardlau]

I've temporarily removed the ability to start new CI jobs and am waiting for the one running test-commit job to complete.
On the FS I've undone #3030 (comment) -- i.e. I've swapped back the jenkins.war file to the version matching what apt thinks is installed (2.361.1 -- we are on 2.346.3) and will attempt to restart Jenkins once no more jobs are running.

# unzip -p /usr/share/java/jenkins.war META-INF/maven/org.jenkins-ci.main/jenkins-war/pom.xml  | gre
p version
<?xml version="1.0" encoding="UTF-8"?>
  <version>2.361.1</version>
      <version>2.361.1</version>
      <version>3044.vb_940a_a_e4f72e</version>
      <version>2.361.1</version>
      <version>2.361.1</version>
      <version>2.361.1</version>
      <version>1.7.36</version>
      <version>[0]</version>

Assuming Jenkins restarts on 2.346.3 I'll look at updating plugins and then attempt an apt update of Jenkins.

[richardlau]

I've checked that the current set of Jenkins plugins are in the daily backup on the backup server (in case we need to go back).

On the CI server:

LC_ALL=C find /var/lib/jenkins/plugins/ -type f -name '*.?pi' -follow -print | sort -d | while read FILENAME; do basename $FILENAME ; unzip -p $FILENAME META-INF/MANIFEST.MF | sort -d | grep 'Short-Name\|Long-Name\|Plugin-Version' ; echo --- ; done > /tmp/before.plugins.txt

and on the backup server

LC_ALL=C find /backup/periodic/daily.0/ci.nodejs.org/plugins/ -type f -name '*.?pi' -follow -print | sort -d | while read FILENAME; do basename $FILENAME ; unzip -p $FILENAME META-INF/MANIFEST.MF | sort -d | grep 'Short-Name\|Long-Name\|Plugin-Version' ; echo --- ; done > /tmp/backup.plugins.txt

and validated with diff that there are no differences.
FWIW gist of the list of plugins: https://gist.github.com/richardlau/85eb9ca0164e2d68fb3635fcdd5842d9

[richardlau]

The update did not go well. Jenkins refused to start -- the GitHub Authentication plugin refused to load due to missing/broken dependencies. Adoptium saw a similar thing when they also tried to update from a similar level of Jenkins. @sxa (🙇‍♂️) and I spent a couple of hours to get us back to a similar state to where we were before attempting the update -- we now still on the same earlier version of Jenkins (2.346.3) but with a slightly upgraded set of plugins.

May try again tomorrow.

[nschonni]

Not for this upgrade, but I wonder if the dockerized version might be easier to upgrade in the future https://github.com/jenkinsci/docker/blob/master/README.md

[richardlau]

The problem we have is plugin dependency issues. Particularly Jenkins wants you to update plugins before updating Jenkins, but some of the plugins cannot be upgraded as they require newer versions of Jenkins to run (in some cases they require a version of Jenkins newer than the most recent LTS version 😞 ).

[richardlau]

I think post-build-status-update has been broken again after today's failed update #3159 (comment) 😞