Add basic timeout functionality to QueryResource

[drcrallen]

• Add a waiting pool that can be used to timeout actions.
• Does not stop queries which have partially returned results. Only stops queries which are completely blocked on the historical level.

[drcrallen]

This is a partial solution for
#1415

A more complete solution will also interrupt the response correctly, but that is a bit more tricky since the response is already building an Ok, I'll probably need to check with @xvrl on how to handle that.

[himanshug]

Is there a reason listeningDecorator(..) is needed and ExecutorService is not enough?

[nishantmonu51]

if i am understanding it correct, its listeningexecutor so that the futures can be registered with QueryWatcher,
see QueryManager.registerQuery()

[drcrallen]

Yes, I needed to register

[himanshug]

got it, thanks

[xvrl]

The main problem I see with adding a second thread pool where queries are waiting, is that we cannot guarantee that resulting yielders will always get closed, some of them may get discarded without being consumed, even if we cancel the future. If we don't close yielders, resources will not get released.

The other issue is that in the event of a large query backlog, we may potentially be creating hundreds of threads, since the pool in unbounded. Creating threads is expensive, so this will cause strain on the system, and compound the backlog even more.

A simpler solution might be to register the queries with the QueryManager before submitting them to the processing pool, and have the query processing ensure that already cancelled queries get interrupted before they even start doing any work as they are taken from the backlog.

[drcrallen]

@xvrl : In the current way of doing things the thread pool will still explode, but it will be the http thread pool, So this pool shouldn't ever be larger than the http pool size.

The yielder not closing is definitely a problem with this approach. But looking at the different sequence.toYielder methods, some of them seem to properly closeout on error and others do not. This seems like a problem with sequences rather than this particular approach.

Having a check with the processing queue is reasonable, but still does not guarantee that the REST method will return in anything resembling the requested timeout.

[xvrl]

@drcrallen if there are toYielder methods with issues, we should fix those for sure.

Having the REST method timeout properly is always going to be tricky with the standard servlet model.

At the router level we handle this by setting the proper timeouts at the async servlet level. Maybe switching to async servlet here would be the right way to do things, since that comes with it's own separate non-blocking pool for I/O handling, and another pool for doing the work.

[drcrallen]

I split the test ( metamx@27e4e1f ) and proposed fix into different commits for easier changes down the line in this PR

[drcrallen]

from AsyncQueryingServelet:

/**
 * This class does async query processing and should be merged with QueryResource at some point
 */

[drcrallen]

Announcer test problems... restarting