couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid

[skeyby]

In our environment CouchDB is sitting behind an Apache HTTPD with mod_proxy_balancer.

Whenever I start a replication toward the HTTPD proxied url I start getting these errors in couch log:

[error] 2019-01-10T15:35:50.396182Z couchdb@10.99.1.1 <0.6123.0> -------- couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid
[error] 2019-01-10T15:35:53.871404Z couchdb@10.99.1.1 <0.14219.1> -------- couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid
[error] 2019-01-10T15:35:53.872080Z couchdb@10.99.1.1 <0.14232.1> -------- couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid
[error] 2019-01-10T15:35:53.889202Z couchdb@10.99.1.1 <0.14219.1> -------- couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid
[error] 2019-01-10T15:35:53.904773Z couchdb@10.99.1.1 <0.14232.1> -------- couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid

If I start the same exact replication pointing directly to couch the error disappears.

The replication seems to work ok except for the fact that it seems to be somewhat slower in the initial kickoff.

HTTPD is injecting a cookie (called BALANCERID) in the answer to handle sticky session so I suppose the error is about that, yet the cookie itself doesn't seems malformed.

Is there something wrong going on or is the log just too verbose?

This is a comparison between direct and proxied url:

abrancatelli@MacBook-Air-di-Andrea ~> curl -v http://10.33.102.50:5984
* Rebuilt URL to: http://10.33.102.50:5984/
*   Trying 10.33.102.50...
* TCP_NODELAY set
* Connected to 10.33.102.50 (10.33.102.50) port 5984 (#0)
> GET / HTTP/1.1
> Host: 10.33.102.50:5984
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Cache-Control: must-revalidate
< Content-Length: 183
< Content-Type: application/json
< Date: Thu, 10 Jan 2019 15:49:57 GMT
< Server: CouchDB/2.3.0 (Erlang OTP/21)
< X-Couch-Request-ID: 308c9970a8
< X-CouchDB-Body-Time: 0
<
{"couchdb":"Welcome","version":"2.3.0","git_sha":"07ea0c7","uuid":"0a959b9b8227188afc2ac26ccdf345a6","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"FreeBSD"}}
* Connection #0 to host 10.33.102.50 left intact

abrancatelli@MacBook-Air-di-Andrea ~> curl -v https://XXXXXX.schema31.it
* Rebuilt URL to: https://XXXXXX.schema31.it/
*   Trying 2.228.74.190...
* TCP_NODELAY set
* Connected to XXXXXX.schema31.it (2.228.74.190) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN= XXXXXX.schema31.it
*  start date: Jan 10 13:00:14 2019 GMT
*  expire date: Apr 10 13:00:14 2019 GMT
*  subjectAltName: host "XXXXXX.schema31.it" matched cert's "XXXXXX.schema31.it"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: XXXXXX.schema31.it
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 10 Jan 2019 15:50:22 GMT
< Server: CouchDB/2.3.0 (Erlang OTP/21)
< Cache-Control: must-revalidate
< Content-Length: 183
< Content-Type: application/json
< X-Couch-Request-ID: 70b4e6a60a
< X-CouchDB-Body-Time: 0
< X-GCloud-FE-Profile: D=60387
< X-GCloud-FE-Identity: cianuro
< X-GCloud-BE-Route: saferm01
< Set-Cookie: BALANCERID=.saferm01; path=/
<
{"couchdb":"Welcome","version":"2.3.0","git_sha":"07ea0c7","uuid":"0a959b9b8227188afc2ac26ccdf345a6","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"FreeBSD"}}
* Connection #0 to host XXXXXX.schema31.it left intact

[wohali]

Do not use sticky sessions with CouchDB, period.

[wohali]

That error is coming from the replicator trying to parse the AuthSession cookie header, if it's missing, the replication fails.

Be sure you are not removing this header.

If you are trying to replicate against 2.2.0, you need to turn off session-based authentication in the replicator, see #1607.

[skeyby]

Do not use sticky sessions with CouchDB, period.

OK, that makes sense - I'm removing it, let's see if the error disappears.

[skeyby]

Do not use sticky sessions with CouchDB, period.

Joan, removing sticky session immediately broke Fauxton (it seems like it broke sessions).

[wohali]

@skeyby That means that you have not set up CouchDB correctly. You need the exact same admin password hashes on every machine - the [admins] line in local.ini must be completely identical on all boxes. As well, all machines need the exact same [couchdb] uuid. Please check.

[wohali]

The uuid bit is documented at https://docs.couchdb.org/en/latest/setup/cluster.html?highlight=uuid

The password hash bit is currently being hashed out (hah) over in #1803 and #1781.

[skeyby]

Thanks Joan, the UUID and Secret were aligned on the two nodes unfortunately the admin users were created before aligning them. 👌🏻

[peterfarrell]

I know this issue is closed, but leaving a breadcrumb for those that come behind me since this is the top search result.

We recently encountered the error described on a CouchDb instance that was replicating from a "primary". We were seeing 100k errors or more per hour. Replication was still working.

couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid

This was CouchDB 3.3.3 which was hosted on OpenShift Container Platform (OCP). We figured out that OCP's router (HAProxy) was the cause as it was adding a cookie to enable "sticky" sessions.

We were able to disable the cookie behavior using an annotation on our route. This is the patch file:

metadata:
  annotations:
    haproxy.router.openshift.io/disable_cookies: "true"
# Cluster route must disable session cookie from OCP HAProxy Router
# Otherwise the secondary CouchDB instance will have many failures in the log file
# couch_replicator_auth_session : Could not parse cookie from response headers cookie_format_invalid

[nickva]

Thanks for reaching out @peterfarrell. It's interesting that the replication still worked. But it does make sense based on the behavior of the replicator. When in the response from the endpoints it sees a new cookie, it will assume it's a refreshed cookie sent by the CouchDB, and it will attempt to parse it (to get the max age) and start using the new cookie. However, if it can't parse it, it will emit an error (to let the user know) and continue.

I guess eventually when the cookie expired, the real cookie from CouchDB endpoints makes it to the replicator, otherwise the replication would stop working eventually?

It's great you found a solution to stop the generation of log errors. Another potential fix might be to disable cookie authentication and use basic auth for the replicator.

https://docs.couchdb.org/en/stable/config/replicator.html#replicator/auth_plugins

[replicator]
auth_plugins = couch_replicator_auth_noop

The downside there is with the upgrade to a stronger hash (kdf) function each request will take longer. So your solution to disable the extraneous cookie is preferable.

[rnewson]

unless that other cookie was also called AuthSession I think we can (and should) make couchdb ignore such things.

[peterfarrell]

Unless otherwise set by an annotation on the route using router.openshift.io/cookie_name="NAMEOFCOOKIE", the name of the cookie is a hashed name of the route:

As indicated in my previous comment, I disabled sticky session routing with haproxy.router.openshift.io/disable_cookies="true". I did not check to see if naming the cookie explictly changed the behavior or stopped the error.

IIRC, I tried setting the auth_plugins to this value first without success however I may have not set it on the primary instance.

[replicator]
auth_plugins = couch_replicator_auth_noop

[nickva]

unless that other cookie was also called AuthSession I think we can (and should) make couchdb ignore such things.

Yeah, good point, that's a bug we should fix. Currently, if we find any cookies we expect they would be AuthSession cookies and fail with an error log if they are not. We should expect other cookies and ignore them. I'll reopen the issue and fix it so we ignore extra cookies.

[peterfarrell]

@nickva seems like a reasonable expectation that additional cookies (non-relevant to CouchDB) could also be present.