introduce /druid/v3 query endpoint that gives query responseContext

[himanshug]

…in response json instead of HTTP header ("X-Druid-Response-Context")

this is primarily required to handle concerns regarding limits to the size of HTTP headers [in jetty and other places]. Currently we have a hack in place to always truncate context ( #2331 ) which makes responseContext somewhat unusable on the client side.

so, to ensure backward compatibility, this PR introduces a new query endpoint "/druid/v3" that has following response format...

{
"result": [ result1, result2,..... ],
"context": { "k1": < value > , .... }
}

Note that backward compatibility can also be achieved by utilizing a new value for HTTP "Accept" header instead of a new endpoint. But, chose new endpoint for "simplicity" (or at least thought it was simpler). I am willing to switch to that if others believe that is better than a new endpoint.

[drcrallen]

can this be in a try with resources block to guarantee it is closed?

[himanshug]

sure, copy/pasta from base class code... not sure what happens if the close is not called in an exception situation. will put additional try-catch in both classes .... or may be refactor a bit more so that this part is also re-used.

only thing different in this class really is not dumping context header and passing it to jsonWriter instead.

[himanshug]

@drcrallen updated with a try-finally

[nishantmonu51]

I believe this needs to be registered for router, historical, RT Index task and RT Nodes as well ?

[himanshug]

@nishantmonu51 this PR is for only enabling the endpoint on broker.

full historical and realtime node support is being added in #3323

i have to admit that I had forgotten about the router, adding router in #3323 as well. (i guess router will be simpler than historical/realtime because router does not deserialize the response and just a pass through )

[drcrallen]

license looks weird, can you re-apply?

[himanshug]

updated, not sure why my IDE license header got messed up.... it seems i need to re-apply in other PRs as well :(

[xvrl]

@himanshug since we're updating the API, I wonder if it would make sense to define whether we always return the results before context. This would allow us to stream results back, as well as include some query stats (or other data that we only know after the fact) as part of the context.

[gianm]

The thing I was concerned about was if the serializing fails before Jackson gets to the Yielder. This test wouldn't verify that (although it does verify a separate equally important thing).

Could you also do a test where the Yielder is just a normal Yielder (perhaps from a ResourceClosingSequence), but the output stream fails on all calls to write? I think that'd be enough to smoke out the case I'm thinking of.

[himanshug]

added another test to verify OutputStream failure, yielder is closed as expected.

[himanshug]

@xvrl we discussed the ordering or result and context in #3319 (diff)
updated the querying documentation with information about new endpoint. I am marking it experimental for now so as to be able to change things in near future if a problem is discovered with the response format proposed here.

[himanshug]

moved to 0.9.3 to unblock 0.9.2

[fjy]

@himanshug there's some merge conflicts

[himanshug]

after some more thought on this recently, I'm planning to have both pre and post context, and, make the response format be...

{
"pre-context" : { "k": < value > , .... }
"result": [ result1, result2,..... ],
"post-context": { "k1": < value > , .... }
}

that can enable the cases where broker sends a query to some historical which does not have all the segments that broker thought it has (due to zookeeper+curator weirdness e.g.). in those cases, query fails currently.
we could envision a change where historical could quickly stream a pre-context with the list of segments it does not have and broker could try other historicals for those segments... in the end query would not have to fail.

[stale]

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[stale]

This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.