introduce mechanism to limit response context size

[himanshug]

druid query response can contain header "X-Druid-Response-Context" which is unbounded in size and can lead to exception like below.

2015-12-16T17:33:21,646 WARN [qtp296974277-44] org.eclipse.jetty.server.HttpChannel - /druid/v2/?pretty
java.io.IOException: Response header too large
    at org.eclipse.jetty.http.HttpGenerator.generateResponse(HttpGenerator.java:400) ~[jetty-http-9.2.5.v20141112.jar:9.2.5.v20141112]
    at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:599) ~[jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
    at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:246) ~[jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
    at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:208) ~[jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
    at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:448) ~[jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
    at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:762) ~[jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
    at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:800) ~[jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
    at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:139) ~[jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
    at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:132) ~[jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
    at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:347) ~[jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
...
Caused by: java.nio.BufferOverflowException
    at java.nio.Buffer.nextPutIndex(Buffer.java:521) ~[?:1.8.0_51]
    at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:169) ~[?:1.8.0_51]
    at org.eclipse.jetty.http.HttpGenerator.putSanitisedValue(HttpGenerator.java:1057) ~[jetty-http-9.2.5.v20141112.jar:9.2.5.v20141112]
    at org.eclipse.jetty.http.HttpGenerator.putTo(HttpGenerator.java:1079) ~[jetty-http-9.2.5.v20141112.jar:9.2.5.v20141112]
    at org.eclipse.jetty.http.HttpGenerator.generateHeaders(HttpGenerator.java:703) ~[jetty-http-9.2.5.v20141112.jar:9.2.5.v20141112]
    at org.eclipse.jetty.http.HttpGenerator.generateResponse(HttpGenerator.java:385) ~[jetty-http-9.2.5.v20141112.jar:9.2.5.v20141112]
    ... 74 more

there should be a mechanism to configure the header size in jetty and also a way to enforce that limit w/o getting jetty exceptions and query failure.

it was exposed in #2108

[drcrallen]

would it be worth investigating http trailer support at all?

[himanshug]

one simple option is to put a check in QueryResource on size of header and truncate any header to something like 4k and log.warn("blah header was truncated").
not sure if there is a way to reliably determine "4k".

[himanshug]

other comments

@drcrallen : There should be a configuration in jetty to increase the allowed header size as well. Currently there's not a great way to configure jetty options within Druid though.

@xvrl : while there is theoretically no limit to http header sizes, I am worried increasing the header size could cause interoperability issues with browsers, proxies, or other http clients querying Druid. Most of those set limits on the size of http headers, which the user may not have much control over.

@cheddar: Fwiw, I think that the "solution" Druid implements for (1) can be as simple "if the content of the query ocntext map is greater than 2kb, log it and replace it with a UUID that can be used to find the log"

[xvrl]

@himanshug @cheddar it might be worthwhile to think of what is necessary vs. nice to have in the response context. Dropping all of it just because one entry doesn't fit seems a bit heavy-handed.

Some pieces of the response context are going to be used programmatically and expected to be present, others are more informational in nature.

One solution would be to say it should be the responsibility of whichever code is adding to the context to not exceed a certain size, and hope we don't have too many things writing to the context.

Another might be to have a separate section of the context that can be dropped and logged if needed.
We already have a query UUID so we can just reuse that in the log.

[himanshug]

@xvrl currently whole response context map is dumped in single header, we can create one "X-Druid-Response-Context-<key>" for each key in the map and truncate individually if they are larger than 4K .
we might end up having multiple headers but i don't think there is a limit as such on number of headers.. in any case number of entries in response context map should be smallish.

[cheddar]

Some pieces of the response context are going to be used programmatically and expected to be present, others are more informational in nature.

@xvrl shouldn't the points of code looking at the context be robust to it not being there? AFAIK, the context is informational at best, not actually a necessary part for functionality. If it is being used as a necessary part of the flow instead of just informational purposes, I think we should re-evaluate its status as an HTTP Header...

[himanshug]

ok, it doesn't matter whether we have one header for each element in the responseContext map ...jetty uses one Buffer to hold all the headers.
best we can do really is to truncate the result of jsonMapper.writeValueAsString(responseContext) to 4k-8k or something. and it should be ok or else query will fail if responseContext accidentally has more stuff than 8k.

[mitch-hulu]

We have starting seeing this on 8.3. Any reason to not port it back?