Clean installing fails on postgresql error

[JustScreaMy]

Hello,
I have recently tried to install the version 2024.12.3 on my cluster using the tutorial with a small changes, but I have always ended on postgresql errors:

postgresql 11:27:33.38 INFO  ==> 
postgresql 11:27:33.38 INFO  ==> Welcome to the Bitnami postgresql container
postgresql 11:27:33.38 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
postgresql 11:27:33.38 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
postgresql 11:27:33.38 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
postgresql 11:27:33.38 INFO  ==> 
postgresql 11:27:33.39 INFO  ==> ** Starting PostgreSQL setup **
postgresql 11:27:33.40 INFO  ==> Validating settings in POSTGRESQL_* env vars..
postgresql 11:27:33.41 INFO  ==> Cleaning stale /bitnami/postgresql/data/postmaster.pid file
postgresql 11:27:33.42 INFO  ==> Loading custom pre-init scripts...
postgresql 11:27:33.43 INFO  ==> Initializing PostgreSQL database...
postgresql 11:27:33.48 INFO  ==> pg_hba.conf file not detected. Generating it...
postgresql 11:27:33.48 INFO  ==> Generating local authentication configuration
postgresql 11:27:33.49 INFO  ==> Deploying PostgreSQL with persisted data...
postgresql 11:27:33.50 INFO  ==> Configuring replication parameters
postgresql 11:27:33.51 INFO  ==> Configuring fsync
postgresql 11:27:33.52 INFO  ==> Configuring synchronous_replication
postgresql 11:27:33.58 INFO  ==> Loading custom scripts...
postgresql 11:27:33.58 INFO  ==> Enabling remote connections
postgresql 11:27:33.59 INFO  ==> ** PostgreSQL setup finished! **

postgresql 11:27:33.60 INFO  ==> ** Starting PostgreSQL **
2025-02-01 11:27:33.674 GMT [1[] LOG:  pgaudit extension initialized
2025-02-01 11:27:33.718 GMT [1[] LOG:  starting PostgreSQL 15.8 on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2025-02-01 11:27:33.719 GMT [1[] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2025-02-01 11:27:33.719 GMT [1[] LOG:  listening on IPv6 address "::", port 5432
2025-02-01 11:27:33.753 GMT [1[] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2025-02-01 11:27:33.852 GMT [96[] LOG:  database system was interrupted; last known up at 2025-02-01 11:27:21 GMT
2025-02-01 11:27:33.945 GMT [96[] LOG:  database system was not properly shut down; automatic recovery in progress
2025-02-01 11:27:33.976 GMT [96[] LOG:  redo starts at 0/1500660
2025-02-01 11:27:33.976 GMT [96[] LOG:  invalid record length at 0/1500710: wanted 24, got 0
2025-02-01 11:27:33.976 GMT [96[] LOG:  redo done at 0/1500698 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
2025-02-01 11:27:34.007 GMT [94[] LOG:  checkpoint starting: end-of-recovery immediate wait
2025-02-01 11:27:34.319 GMT [94[] LOG:  checkpoint complete: wrote 3 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.101 s, sync=0.001 s, total=0.343 s; sync files=2, longest=0.001 s, average=0.001 s; distance=0 kB, estimate=0 kB
2025-02-01 11:27:43.355 GMT [107[] FATAL:  password authentication failed for user "authentik"
2025-02-01 11:27:43.355 GMT [107[] DETAIL:  Role "authentik" does not exist.
    Connection matched pg_hba.conf line 1: "host     all             all             0.0.0.0/0               md5"
2025-02-01 11:27:44.363 GMT [108[] FATAL:  password authentication failed for user "authentik"
2025-02-01 11:27:44.363 GMT [108[] DETAIL:  Role "authentik" does not exist.
    Connection matched pg_hba.conf line 1: "host     all             all             0.0.0.0/0               md5"

My Values.yaml looks like this:

postgresql:
  enabled: true
  auth:
    password: "myVerySecretPassword"

authentik:
  secret_key: "myVerySecretSecret"

  postgresql:
    password: "myVerySecretPassword"

redis:
  enabled: true

server:
  ingress:
    ingressClassName: nginx
    enabled: true
    hosts:
      - idp.example.com
    annotations:
      nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
      nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
      cert-manager.io/cluster-issuer: cloudflare-issuer
    tls:
      - hosts: 
        - idp.example.com
        secretName: authentik-tls

[JustScreaMy]

I wasnt able to reproduce this errors locally on Minikube, so it might be something wrong with my environment.
Im running k3s cluster with csi-driver-nfs for PVC, but i dont see errors anywhere.

[rissson]

From the following logs, this doesn't seem like a clean install

2025-02-01 11:27:33.852 GMT [96[] LOG:  database system was interrupted; last known up at 2025-02-01 11:27:21 GMT
2025-02-01 11:27:33.945 GMT [96[] LOG:  database system was not properly shut down; automatic recovery in progress

[JustScreaMy]

I have noticed that now, but i have tried it recently again and these are newer logs with all PVCs deleted, and new namespace used

Defaulted container "postgresql" out of: postgresql, init-chmod-data (init)
postgresql 21:51:39.53 INFO  ==>
postgresql 21:51:39.53 INFO  ==> Welcome to the Bitnami postgresql container
postgresql 21:51:39.53 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
postgresql 21:51:39.53 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
postgresql 21:51:39.53 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
postgresql 21:51:39.54 INFO  ==>
postgresql 21:51:39.54 DEBUG ==> Configuring libnss_wrapper...
postgresql 21:51:39.54 DEBUG ==> Copying files from /opt/bitnami/postgresql/conf.default to /opt/bitnami/postgresql/conf
postgresql 21:51:39.55 INFO  ==> ** Starting PostgreSQL setup **
postgresql 21:51:39.56 INFO  ==> Validating settings in POSTGRESQL_* env vars..
postgresql 21:51:39.56 INFO  ==> Loading custom pre-init scripts...
postgresql 21:51:39.56 INFO  ==> Initializing PostgreSQL database...
postgresql 21:51:39.57 DEBUG ==> Copying files from /bitnami/postgresql/conf to /opt/bitnami/postgresql/conf
postgresql 21:51:39.57 DEBUG ==> Ensuring expected directories/files exist...
postgresql 21:51:39.73 INFO  ==> pg_hba.conf file not detected. Generating it...
postgresql 21:51:39.73 INFO  ==> Generating local authentication configuration
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /bitnami/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... %

And then it started again

postgresql 21:53:38.35 INFO  ==>
postgresql 21:53:38.35 INFO  ==> Welcome to the Bitnami postgresql container
postgresql 21:53:38.35 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
postgresql 21:53:38.35 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
postgresql 21:53:38.35 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
postgresql 21:53:38.35 INFO  ==>
postgresql 21:53:38.36 DEBUG ==> Configuring libnss_wrapper...
postgresql 21:53:38.36 DEBUG ==> Copying files from /opt/bitnami/postgresql/conf.default to /opt/bitnami/postgresql/conf
postgresql 21:53:38.37 INFO  ==> ** Starting PostgreSQL setup **
postgresql 21:53:38.38 INFO  ==> Validating settings in POSTGRESQL_* env vars..
postgresql 21:53:38.38 INFO  ==> Cleaning stale /bitnami/postgresql/data/postmaster.pid file
postgresql 21:53:38.41 INFO  ==> Loading custom pre-init scripts...
postgresql 21:53:38.41 INFO  ==> Initializing PostgreSQL database...
postgresql 21:53:38.41 DEBUG ==> Copying files from /bitnami/postgresql/conf to /opt/bitnami/postgresql/conf
postgresql 21:53:38.41 DEBUG ==> Ensuring expected directories/files exist...
postgresql 21:53:38.46 INFO  ==> pg_hba.conf file not detected. Generating it...
postgresql 21:53:38.47 INFO  ==> Generating local authentication configuration
postgresql 21:53:38.47 INFO  ==> Deploying PostgreSQL with persisted data...
postgresql 21:53:38.48 INFO  ==> Configuring replication parameters
postgresql 21:53:38.50 INFO  ==> Configuring fsync
postgresql 21:53:38.50 INFO  ==> Configuring synchronous_replication
postgresql 21:53:38.61 INFO  ==> Loading custom scripts...
postgresql 21:53:38.61 INFO  ==> Enabling remote connections
postgresql 21:53:38.62 INFO  ==> ** PostgreSQL setup finished! **

postgresql 21:53:38.63 INFO  ==> ** Starting PostgreSQL **
2025-02-05 21:53:38.713 GMT [1] LOG:  pgaudit extension initialized
2025-02-05 21:53:38.759 GMT [1] LOG:  starting PostgreSQL 15.8 on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2025-02-05 21:53:38.760 GMT [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2025-02-05 21:53:38.760 GMT [1] LOG:  listening on IPv6 address "::", port 5432
2025-02-05 21:53:38.792 GMT [1] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2025-02-05 21:53:38.914 GMT [100] LOG:  database system was interrupted; last known up at 2025-02-05 21:53:25 GMT
2025-02-05 21:53:39.016 GMT [100] LOG:  database system was not properly shut down; automatic recovery in progress
2025-02-05 21:53:39.048 GMT [100] LOG:  redo starts at 0/1500660
2025-02-05 21:53:39.048 GMT [100] LOG:  invalid record length at 0/1500710: wanted 24, got 0
2025-02-05 21:53:39.048 GMT [100] LOG:  redo done at 0/1500698 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
2025-02-05 21:53:39.080 GMT [98] LOG:  checkpoint starting: end-of-recovery immediate wait
2025-02-05 21:53:39.459 GMT [98] LOG:  checkpoint complete: wrote 3 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.123 s, sync=0.001 s, total=0.411 s; sync files=2, longest=0.001 s, average=0.001 s; distance=0 kB, estimate=0 kB
2025-02-05 21:53:39.705 GMT [1] LOG:  database system is ready to accept connections
2025-02-05 21:53:48.447 GMT [111] FATAL:  password authentication failed for user "authentik"
2025-02-05 21:53:48.447 GMT [111] DETAIL:  Role "authentik" does not exist.
        Connection matched pg_hba.conf line 1: "host     all             all             0.0.0.0/0               md5"
2025-02-05 21:53:49.455 GMT [112] FATAL:  password authentication failed for user "authentik"
2025-02-05 21:53:49.455 GMT [112] DETAIL:  Role "authentik" does not exist.
        Connection matched pg_hba.conf line 1: "host     all             all             0.0.0.0/0               md5"

And when trying to connect manually to postgres user, it says

postgresql 2025-02-05 21:31:47.405 GMT [293[] FATAL:  password authentication failed for user "postgres"
postgresql 2025-02-05 21:31:47.405 GMT [293[] DETAIL:  User "postgres" has no password assigned.
postgresql     Connection matched pg_hba.conf line 3: "local    all             all                                     md5"

[only-a-user]

I encountered the same issue. For me it seems like the password set for postgres is disregarded, as the authentik-postgresql secret uses an entirely different password for the authentik account.

A quick and dirty workaround is entering the shell of the postgresql pod and then logging into the postgres user with this command:

psql -U postgres

Quick side note: Have the password for the postgres user ready, in the aforementioned secret it is set as postgres-postgres-password.

Then execute this command:

ALTER USER authentik WITH PASSWORD 'YourSuperSecretPassword';