fix cache populate incorrect content when numBackgroundThreads>1

[kaijianding]

The origin code can cause:

1. concurrency issue
2. sequence order and cacheResults order maybe different, eg: for timeSeries query,
   the sequence order is

[{
  "timestamp" : "2017-02-16T09:00:48.000+08:00",
  "result" : {
    "__result" : 1.0
  }
}, {
  "timestamp" : "2017-02-16T09:01:04.000+08:00",
  "result" : {
    "__result" : 1.0
  }
}, {
  "timestamp" : "2017-02-16T09:01:20.000+08:00",
  "result" : {
    "__result" : 1.0
  }
}]

but the cacheResults order can be

[{
  "timestamp" : "2017-02-16T09:00:48.000+08:00",
  "result" : {
    "__result" : 1.0
  }
}, {
  "timestamp" : "2017-02-16T09:01:20.000+08:00",
  "result" : {
    "__result" : 1.0
  }
}, {
  "timestamp" : "2017-02-16T09:01:04.000+08:00",
  "result" : {
    "__result" : 1.0
  }
}]

Basically, this issue is caused by the cacheResults.add() called order is not promised as same as backgroundExecutorService.submit() called order when numBackgroundThreads>1

This difference can cause incorrect merge sort result on broker.

[gianm]

LGTM, just a question on the tests.

thx for the fix @kaijianding!

[gianm]

Is it possible to do this with a CountDownLatch or similar so we are guaranteed it finishes?

[kaijianding]

Modified, @gianm

[drcrallen]

I believe there's a Futures.allAsList method in guava

[kaijianding]

Modified, @drcrallen

[drcrallen]

At one point this had problems with HLL buffers, I don't remember what the issue was off hand. I think the buffer was being released before the cache was calculated or something.

[KurtYoung]

please add a new line after this.

[kaijianding]

Modefied, @KurtYoung

[drcrallen]

I'm on my phone right now but doesn't await just return a boolean and not actually throw an error on timeout?

[kaijianding]

Modified, @drcrallen

[nishantmonu51]

👍