Fix fetch of task location in SpecificTaskServiceLocator

[kfaraz]

Description

The task status API was improved in #15724 to serve task statuses from the Overlord memory.
But on older Overlord versions, this API would return an unknown task status location causing failures in inter-task communications.

This was later fixed in #16227 but that introduced another bug described below. This bug is typically reproducible in MSQ controller tasks but may occur in native batch ingestion as well.

• The SpecificTaskServiceLocator first calls the multi-task status API /druid/indexer/v1/taskStatus to determine the location of a task.
• This API returns an unknown task location on old Overlord versions
• The SpecificTaskServiceLocator then falls back to calling the single task status API /druid/indexer/v1/task/{taskId}/status
• The problem is that the second API is invoked in a synchronous manner in the callback of the first future while holding a lock
• If there are enough MSQ workers, this could cause all the ServiceClientFactory threads to be stuck waiting to get back a task location. But to fetch the task location, we need one of the ServiceClientFactory threads.

Fix

• Invoke the single task status API in an async manner
• Ensure that success/failure callbacks are quick and do not keep threads blocked

Testing

Local setup to reproduce issue:

• Build Druid on commit f643abdad9 and run overlord, coordinator, broker
• Build Druid on commit f82cc34e5b and run middle manager with 10 task slots
• Submit a large MSQ query with 6 workers + 1 controller
• Verify that controller remains stuck and doesn't make progress since all ServiceClientFactory threads are busy fetching task location

Local setup to verify fix:

• Build Druid on commit f643abdad9 and run overlord, coordinator, broker
• Checkout Druid on commit f82cc34e5b
• Apply the changes in this PR
• Build Druid and run middle manager with 10 task slots
• Submit a large MSQ query with 6 workers + 1 controller
• Verify that the controller does not get stuck and live ingestion stats are reported as expected.

[gianm]

Generally LGTM other than the one exception handling thing.

[gianm]

Should resolve pendingFuture in this case, I think. It would end up abandoned if an exception is actually thrown here.

[kfaraz]

Thanks for catching this. Fixed.