DB migration jobs fail on PostgreSQL14.3 for Mastodon 4.3.0

[hardillb]

kubectl -n mastodon logs mastodon-db-migrate-qjlsp
I, [2024-10-08T22:13:18.132471 #1]  INFO -- : Migrating to AddURLToPreviewCardsStatuses (20231006183200)
== 20231006183200 AddURLToPreviewCardsStatuses: migrating =====================
[strong_migrations] DANGER: No lock timeout set
-- add_column(:preview_cards_statuses, :url, :string)
   -> 0.0656s
== 20231006183200 AddURLToPreviewCardsStatuses: migrated (0.0739s) ============

I, [2024-10-08T22:13:18.239826 #1]  INFO -- : Migrating to AddIndexToWebauthnCredentialsUserIdNickname (20231018192110)
== 20231018192110 AddIndexToWebauthnCredentialsUserIdNickname: migrating ======
[strong_migrations] DANGER: No lock timeout set
rake aborted!
StandardError: An error has occurred, all later migrations canceled: (StandardError)

=== Dangerous operation detected #strong_migrations ===

Adding an index concurrently can cause silent data corruption in Postgres 14.0 to 14.3.
Upgrade Postgres before adding new indexes, or wrap this step in a safety_assured { ... } block
to accept the risk.

/opt/mastodon/db/migrate/20231018192110_add_index_to_webauthn_credentials_user_id_nickname.rb:31:in `add_index_to_table'
/opt/mastodon/db/migrate/20231018192110_add_index_to_webauthn_credentials_user_id_nickname.rb:7:in `up'
/usr/local/bundle/bin/bundle:25:in `load'
/usr/local/bundle/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

Trying to upgrade the dependencies also fails as PostgreSQL16 won't open the data directory of PostgreSQL 14 to upgrade

[renchap]

You need to update to the latest patch release of Postgresql 14 (14.13). This can be done in place, without upgrading your pg data directory.

If you want to update to a more recent postgres major version, then you need to follow the postgres upgrade procedure, that involves either a dump, or using pg_upgrade.

[hardillb]

@renchap and how do I do that with the version of PostgreSQL that was installed by the helmchart?

Also the release notes for 4.3.0 say that they support for PostgreSQL 12 up

[renchap]

@renchap and how do I do that with the version of PostgreSQL that was installed by the helmchart?

You probably should update postgresql.image.tag to use a more recent image. I can no check right now, but I can look into it tomorrow

Also the release notes for 4.3.0 say that they support for PostgreSQL 12 up

Yes, we missed that one of our dependencies has a specific check for versions 14.0 to 14.3, as they have a known data corruption bug. We will update the release notes tomorrow