Requests return 400 Bad Request when URL length exceeds 1460 characters

[dianabarsan]

When using chttpd.server_option = [{recbuf, undefined}], requests with URL length exceeding 1459 characters fail with 400 Bad Request with no error in the logs. (The relevant URL length excludes protocol and userinfo).

Example of failing request:

curl -v http://127.0.0.1:5984/_users/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

URL length is 1467 (1460 excluding protocol and userinfo)

Expected Behavior

Should return:

Host: 127.0.0.1:5984
User-Agent: curl/7.58.0
Accept: */*
 
HTTP/1.1 404 Object Not Found
Cache-Control: must-revalidate
Content-Length: 41
Content-Type: application/json
Date: Thu, 13 Dec 2018 13:18:50 GMT
Server: CouchDB/2.3.0 (Erlang OTP/20)
X-Couch-Request-ID: 0b4ad7148f
X-CouchDB-Body-Time: 0 
{"error":"not_found","reason":"missing"}

Current Behavior

Actually returns:

HTTP/1.1
Host: 127.0.0.1:5984
User-Agent: curl/7.58.0
Accept: */*
 
HTTP/1.1 400 Bad Request
Connection: close
Content-Length: 0
Date: Thu, 13 Dec 2018 13:15:09 GMT
Server: MochiWeb/1.0 (Any of you quaids got a smint?) 

Example of successful request:

curl -v http://127.0.0.1:5984/_users/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

URL length is 1466 (1459 excluding protocol and userinfo).

Possible Cause

It appears that, by not setting a recbuf value, mochiweb_socket_server sockets end up with a buffer size of 1460 (the user-level buffer - http://erlang.org/doc/man/inet.html), which causes this limitation (recbuf and sndbuf are indeed defaulting to much larger values).
This default value of 1460 seems to be coming from Erl inet_drv.c and initialized here.

Steps to Reproduce

Install CouchDB 2.3 (or build from master) and run with default configuration (specifically chttpd.server_option = [{recbuf, undefined}]).
Execute a request with a URL length (excluding protocol and authority) exceeding 1460 characters (method or endpoint appear to be irrelevant).

Context

We are using PouchDB to interact with our CouchDB database and ran into this error with view queries. PouchDB does issue a POST request instead of a GET, but only when the query string exceeds 2000 characters (see here).

Your Environment

• Version used: 2.3.0, master
• Using: Chrome, cURL, wget
• Operating system: Ubuntu 18, Ubuntu 16 (desktop)

Even if it is decided that this is desired behavior, I believe this change should at least be documented.

[wohali]

Hi @dianabarsan , thanks for the report.

This is definitely not intended - we'll take a closer look. See #1409 for the history here; we can't directly just revert the change because the other use case will be impacted, so this may take a bit of research to resolve sufficiently.

[wohali]

http://erlang.org/pipermail/erlang-questions/2011-June/059571.html
ninenines/cowboy#3
ninenines/cowboy@29e71cf

We're weighing our options now. Currently hoping improving mochiweb to pass [{buffer, 8192}] usefully thru one or another setting in *.ini or similar would be enough. The cowboy solution is going to be more reliable in the long run, but is a fairly large retrofit to mochiweb.

[nickva]

Hi @dianabarsan,

Thank you for the bug report and excellent analysis.

You're absolutely correct, chttpd.server_option = [{recbuf, undefined}]). ends up not setting recbuf value, which means Erlang userland buffer gets a default value of 1460 (when recbuf is set so is the userland buffer).

This buffer size limit combined with a long-standing bug in Erlang's {packet, http} parser ends crashing the socket receive code with an emsgsize error.

It's not possible to set the buffer size in mochiweb currently, so I will make a PR request to fix that.

Notice:

30> f(), SockInfo = fun(S) -> inet:getopts(S, [recbuf, buffer]) end, {ok, LS} = gen_tcp:listen(0, [{recbuf, 8192}]), LRes = SockInfo(LS).
{ok,[{recbuf,8192},{buffer,8192}]}

31> f(), SockInfo = fun(S) -> inet:getopts(S, [recbuf, buffer]) end, {ok, LS} = gen_tcp:listen(0, []), LRes = SockInfo(LS).
{ok,[{recbuf,131072},{buffer,1460}]}

[nickva]

pull request:

mochi/mochiweb#208

[dianabarsan]

Hi @wohali , @nickva

Thank you very much for the quick responses and for the additional context!
The Erlang http packet parser bug was the missing piece in my understanding of why this is happening.

We will be updating our configs to set a recbuf value for 2.3.0 installs.

[wohali]

@dianabarsan Be advised that this may adversely affect your attachment performance (if you use attachments), see #1409.

We are considering a 2.3.1 fix for this bug since it's rather a surprising regression.

[wohali]

@nickva I'm not sure that the fix in mochi/mochiweb#208 is going to satisfy the situation in #1409.

Basically, the patch as written I think only sets the buffer to 8192 if recbuf is undefined. This means if you need >8k of headers you have no choice but to peg recbuf higher, which causes the issue seen in #1409. I don't think it's fair to tradeoff a fix for this with poor attachment performance again.

Ideally we shouldn't be using a fixed 8k value in mochiweb but instead calling getopts to figure out the OS's window size (at least initially) and use that to set buffer's size. I know you had runtime reservations about the performance of inserting a getopts call with every socket open but we should characterize this.

An alternative without going back to mochiweb would be to pass in {buffer, BIGNUM} . I don't know if your patch to mochiweb allows passing buffer9 in now as part of server_options, does it? If so this could be a documented workaround; based on testing on Linux the default was like 50k on my system, and 50k of headers should be sufficient. So in this case, we'd tell people to leave recbuf as undefined and just pass in {buffer, 100000} or so. As long as buffer >= recbuf you're fine, and we can provide the remsh command to determine your actual recbuf value as set by the OS.

But the only real fix for this (arbitrary header length + undefined recbuf) is to stop using {packet, http} and allowing multiple recvs on the socket to get all of the header content if necessary (with appropriate limits on the # of headers and max length for each) which is a MASSIVE change to mochiweb and no one is volunteering to write it. So I'm hoping the tradeoff above is sufficient.

/cc @kocolosk @davisp

[wohali]

@nickva bump, we need to solve this, see #1843. Any comments on my proposal?