CookieAuth works only on one node

[aeytom]

Description

I have a 3-node cluster deployed via helm chart 4.3.1. Connect to the cluster is via the provided K8s services. This results in a round-robin access. Cookie-auth is the choosen auth method. Further requests responds often with 403 responses.

Steps to Reproduce

• setup a 3-node cluster using helm
• Open a session: `curl -v http://couchdb:5984/_session -d 'name=admin&password=…' -H 'Accept: application/json'
• Check session on all nodes with AuthSession from response:
  • `curl -v -H 'Cookie: AuthSession=…' http://couchdb-0.couchdb:5984/_session -H 'Accept: application/json'
  • `curl -v -H 'Cookie: AuthSession=…' http://couchdb-1.couchdb:5984/_session -H 'Accept: application/json'
  • `curl -v -H 'Cookie: AuthSession=…' http://couchdb-2.couchdb:5984/_session -H 'Accept: application/json'
• Only one response will {"ok":true,"userCtx":{"name":"admin","roles":["_admin"]},"info":{"authentication_handlers":["cookie","default"],"authenticated":"cookie"}}

Expected Behaviour

All nodes shouldt accept the same AuthSection.

Your Environment

• CouchDB version used: 3.3.2 via helm chart 4.3.1

• all config files on all nodes are equal with exception of ./local.d/docker.ini with different admins.admin pbkdf2 string.

• chttpd_auth.secret and couchdb.uuid are equal on all nodes.

root@cm-prod-couchdb-0:/# cd /opt/couchdb/etc/
root@cm-prod-couchdb-0:/opt/couchdb/etc# find -type f|xargs sha256sum
67c8a6739efe565e9c92e4ecd3700900a809ae52969ce126b436cfcd9d164e68  ./default.ini
da9f783a11c1324b76b10673213489d673653ccb8db243b58bb8078fc5bd030d  ./local.d/README
94e8f2744f9fea8e60f65ec1d5815dc3ca8dc3543ab53f3c3c5d031b9abf5f2a  ./local.d/docker.ini
ab81c1dfd2cdc06a868549bf64027f7c44e994c33110ec8800fc6f7d9945c8e7  ./vm.args
94026bdb2d351d32982edab418d8796d1a341416981d8ef0a1e0543448508d49  ./local.ini
bb8e82668350953651931d51535b72b8a06ddc074e2cca50bcc42fe1455a9d31  ./default.d/seedlist.ini
f4da5e74e9a580aeaf2aee8beb3b1241a6a1209f85c323fbfa50e8c7a89b4c25  ./default.d/chart.ini

root@cm-prod-couchdb-0:/opt/couchdb/etc# cat ./local.d/docker.ini

[admins]
admin = -pbkdf2-…,…,10

[chttpd_auth]
secret = …



root@cm-prod-couchdb-1:/# cd /opt/couchdb/etc/
root@cm-prod-couchdb-1:/opt/couchdb/etc# find -type f|xargs sha256sum
67c8a6739efe565e9c92e4ecd3700900a809ae52969ce126b436cfcd9d164e68  ./default.ini
da9f783a11c1324b76b10673213489d673653ccb8db243b58bb8078fc5bd030d  ./local.d/README
39260c1ca518f21c6e5d9294e8a10a8fe14f6ad35c722a6d3c3d7eceb90c46ff  ./local.d/docker.ini
ab81c1dfd2cdc06a868549bf64027f7c44e994c33110ec8800fc6f7d9945c8e7  ./vm.args
94026bdb2d351d32982edab418d8796d1a341416981d8ef0a1e0543448508d49  ./local.ini
bb8e82668350953651931d51535b72b8a06ddc074e2cca50bcc42fe1455a9d31  ./default.d/seedlist.ini
f4da5e74e9a580aeaf2aee8beb3b1241a6a1209f85c323fbfa50e8c7a89b4c25  ./default.d/chart.ini

root@cm-prod-couchdb-1:/opt/couchdb/etc# cat ./local.d/docker.ini

[admins]
admin = -pbkdf2-…,…,10

[chttpd_auth]
secret = …



root@cm-prod-couchdb-2:/# cd /opt/couchdb/etc/
root@cm-prod-couchdb-2:/opt/couchdb/etc# find -type f|xargs sha256sum
67c8a6739efe565e9c92e4ecd3700900a809ae52969ce126b436cfcd9d164e68  ./default.ini
da9f783a11c1324b76b10673213489d673653ccb8db243b58bb8078fc5bd030d  ./local.d/README
9e722492fcbc5d1e0be393ae70da99c7830cf955f044bfa8f2f25bf2eb5b7801  ./local.d/docker.ini
ab81c1dfd2cdc06a868549bf64027f7c44e994c33110ec8800fc6f7d9945c8e7  ./vm.args
94026bdb2d351d32982edab418d8796d1a341416981d8ef0a1e0543448508d49  ./local.ini
bb8e82668350953651931d51535b72b8a06ddc074e2cca50bcc42fe1455a9d31  ./default.d/seedlist.ini
f4da5e74e9a580aeaf2aee8beb3b1241a6a1209f85c323fbfa50e8c7a89b4c25  ./default.d/chart.ini

root@cm-prod-couchdb-2:/opt/couchdb/etc# cat ./local.d/docker.ini

[admins]
admin = -pbkdf2-…,…,10

[chttpd_auth]
secret = …


./default.ini:[vendor]
./default.ini:name = The Apache Software Foundation
./default.ini:
./default.ini:[couchdb]
./default.ini:uuid = 
./default.ini:database_dir = ./data
./default.ini:view_index_dir = ./data
./default.ini:
./default.ini:[purge]
./default.ini:
./default.ini:[couchdb_engines]
./default.ini:couch = couch_bt_engine
./default.ini:
./default.ini:[process_priority]
./default.ini:
./default.ini:[cluster]
./default.ini:
./default.ini:[chttpd]
./default.ini:port = 5984
./default.ini:bind_address = 127.0.0.1
./default.ini:
./default.ini:[couch_peruser]
./default.ini:
./default.ini:[httpd]
./default.ini:port = 5986
./default.ini:bind_address = 127.0.0.1
./default.ini:
./default.ini:[ssl]
./default.ini:
./default.ini:[chttpd_auth]
./default.ini:
./default.ini:hash_algorithms = sha256, sha
./default.ini:
./default.ini:[couch_httpd_auth]
./default.ini:authentication_db = _users
./default.ini:
./default.ini:[csp]
./default.ini:
./default.ini:[cors]
./default.ini:
./default.ini:[x_frame_options]
./default.ini:
./default.ini:[native_query_servers]
./default.ini:
./default.ini:[query_server_config]
./default.ini:
./default.ini:[mango]
./default.ini:
./default.ini:[indexers]
./default.ini:couch_mrview = true
./default.ini:
./default.ini:[feature_flags]
./default.ini:partitioned||* = true
./default.ini:
./default.ini:[uuids]
./default.ini:
./default.ini:[attachments]
./default.ini:
./default.ini:[replicator]
./default.ini:
./default.ini:[replicator.shares]
./default.ini:
./default.ini:[log]
./default.ini:
./default.ini:[stats]
./default.ini:
./default.ini:[smoosh]
./default.ini:
./default.ini:state_dir = ./data
./default.ini:
./default.ini:[ioq]
./default.ini:
./default.ini:[ioq.bypass]
./default.ini:
./default.ini:[dreyfus]
./default.ini:
./default.ini:[reshard]
./default.ini:
./default.ini:[prometheus]
./default.ini:additional_port = false
./default.ini:bind_address = 127.0.0.1
./default.ini:port = 17986
./default.ini:
./default.ini:[view_upgrade]
./default.ini:
./default.ini:[custodian]
./local.d/docker.ini:
./local.d/docker.ini:[admins]
./local.d/docker.ini:admin = -pbkdf2-…,…,10
./local.d/docker.ini:
./local.d/docker.ini:[chttpd_auth]
./local.d/docker.ini:secret = …
./local.ini:
./local.ini:[couchdb]
./local.ini:
./local.ini:[couch_peruser]
./local.ini:
./local.ini:[chttpd]
./local.ini:
./local.ini:[httpd]
./local.ini:
./local.ini:[ssl]
./local.ini:
./local.ini:[vhosts]
./local.ini:
./local.ini:[admins]
./default.d/seedlist.ini:[cluster]
./default.d/seedlist.ini:seedlist = couchdb@cm-prod-couchdb-0.cm-prod-couchdb.corporate-contentmaschine.svc.cluster.local,couchdb@cm-prod-couchdb-1.cm-prod-couchdb.corporate-contentmaschine.svc.cluster.local,couchdb@cm-prod-couchdb-2.cm-prod-couchdb.corporate-contentmaschine.svc.cluster.local
./default.d/chart.ini:[chttpd]
./default.d/chart.ini:bind_address = any
./default.d/chart.ini:require_valid_user = false
./default.d/chart.ini:
./default.d/chart.ini:[couchdb]
./default.d/chart.ini:uuid = …
./default.d/chart.ini:
./default.d/chart.ini:[log]
./default.d/chart.ini:level = error
./default.d/chart.ini:
./default.d/chart.ini:[smoosh]
./default.d/chart.ini:db_channels = ratio_dbs
./default.d/chart.ini:view_channels = ratio_views
./default.d/chart.ini:
./default.d/chart.ini:[smoosh.ratio_dbs]
./default.d/chart.ini:from = 20:00
./default.d/chart.ini:min_priority = 2.0
./default.d/chart.ini:priority = ratio
./default.d/chart.ini:to = 06:00
./default.d/chart.ini:
./default.d/chart.ini:[smoosh.ratio_views]
./default.d/chart.ini:from = 20:00
./default.d/chart.ini:min_priority = 2.0
./default.d/chart.ini:priority = ratio
./default.d/chart.ini:to = 06:00




www-data@cm-prod-r4-contentmachine-cms-554c944dfd-669bv:~/contentmachine$ curl -v -H 'Cookie: AuthSession=YWRtaW46NjQ1QjcxMDk6BrOyT1KPV8l9jRQM05uxryiREP_MA_2h2B-w3rdcuC0' http://cm-prod-couchdb-0.cm-prod-couchdb:5984/_session -H 'Accept: application/json'
…
< HTTP/1.1 200 OK
< Cache-Control: must-revalidate
< Content-Length: 103
< Content-Type: application/json
< Date: Wed, 10 May 2023 10:26:24 GMT
< Server: CouchDB/3.3.2 (Erlang OTP/24)
< 
{"ok":true,"userCtx":{"name":null,"roles":[]},"info":{"authentication_handlers":["cookie","default"]}}
* Connection #0 to host cm-prod-couchdb-0.cm-prod-couchdb left intact
www-data@cm-prod-r4-contentmachine-cms-554c944dfd-669bv:~/contentmachine$ curl -v -H 'Cookie: AuthSession=YWRtaW46NjQ1QjcxMDk6BrOyT1KPV8l9jRQM05uxryiREP_MA_2h2B-w3rdcuC0' http://cm-prod-couchdb-1.cm-prod-couchdb:5984/_session -H 'Accept: application/json'
…
< HTTP/1.1 200 OK
< Cache-Control: must-revalidate
< Content-Length: 103
< Content-Type: application/json
< Date: Wed, 10 May 2023 10:26:33 GMT
< Server: CouchDB/3.3.2 (Erlang OTP/24)
< 
{"ok":true,"userCtx":{"name":null,"roles":[]},"info":{"authentication_handlers":["cookie","default"]}}
* Connection #0 to host cm-prod-couchdb-1.cm-prod-couchdb left intact
www-data@cm-prod-r4-contentmachine-cms-554c944dfd-669bv:~/contentmachine$ curl -v -H 'Cookie: AuthSession=YWRtaW46NjQ1QjcxMDk6BrOyT1KPV8l9jRQM05uxryiREP_MA_2h2B-w3rdcuC0' http://cm-prod-couchdb-2.cm-prod-couchdb:5984/_session -H 'Accept: application/json'
… 
< HTTP/1.1 200 OK
< Cache-Control: must-revalidate
< Content-Length: 139
< Content-Type: application/json
< Date: Wed, 10 May 2023 10:26:40 GMT
< Server: CouchDB/3.3.2 (Erlang OTP/24)
< Set-Cookie: AuthSession=YWRtaW46NjQ1QjcxNjA6OidYhd96K9-iJt7sYLa5PRETOd5NJf1zhBetSIO5PkQ; Version=1; Expires=Wed, 10-May-2023 10:36:40 GMT; Max-Age=600; Path=/; HttpOnly
< 
{"ok":true,"userCtx":{"name":"admin","roles":["_admin"]},"info":{"authentication_handlers":["cookie","default"],"authenticated":"cookie"}}
* Connection #0 to host cm-prod-couchdb-2.cm-prod-couchdb left intact
www-data@cm-prod-r4-contentmachine-cms-554c944dfd-669bv:~/contentmachine$ 

[willholley]

This is the expected behaviour with the default Helm chart values.

It occurs because each when deploying the helm chart, the admin password is typically specified in plaintext. Each CouchDB node (pod) then hashes the admin password independently on first use, overwriting the plaintext password internally.

CouchDB's cookie authentication relies on the password hash being the same on each database node, which it is not in this case.

You have a few options:

1. In the Helm chart values, set the admin hash explicitly (see https://sleeplessbeastie.eu/2020/03/13/how-to-generate-password-hash-for-couchdb-administrator/) for how to generate this.
2. Use basic auth only for the admin user. That is, create normal CouchDB users for interactive use.

[aeytom]

Thank you for your answer and solution options.

I will try opton one.
Did you why is not the chttpd_auth.secret is used as auth base?

The secret token is used for Proxy Authentication and for Cookie Authentication.

[willholley]

@aeytom you can see the implementation at

couchdb/src/couch/src/couch_httpd_auth.erl

Line 363 in 85e1fa7

VerifyHash = fun(HashAlg) ->

. As I understand it, chttpd_auth.secret is used, but is combined with the salt from the user's password hash which, in this case, varies by node.

[big-r81]

Hey,

the secret needs to be the same on each node, details at section 2.2.3 (Point 4 & 5).

I will try opton one. Did you why is not the chttpd_auth.secret is used as auth base?

It is used for Cookie and Proxy authentication!

couchdb/src/couch/src/couch_httpd_auth.erl

Line 350 in 85e1fa7

case chttpd_util:get_chttpd_auth_config("secret") of