Potential deadlock condition in Appenderator

[pjain1]

• persistExecutor submits mergeAndPush job to pushExecutor here after completing the persist.
• pushExecutor submits abandon segment job to persistExecutor here

pushExecutor uses a blocking queue with 1 capacity and if persistExecutor either uses Synchronous Queue (which it does in case of KafkaIndexTask) or queue with capacity 1 then deadlock can happen. This would be more common in the new KafkaIndexingService which will support incremental handoff as handoffs (which leads to abandoning of segments) for a sequence can interleave with pushing of segments for another sequence.

This can probably be fixed by not using pushBarrier instead using some form of Atomic counter to keep track of existing pushes before abandoning segments. Any thoughts ?

[jihoonson]

I think it's a good idea. Maybe a new thread like abandonExecutor is needed to avoid interleaving?

[pjain1]

May be a new executor can be used as you suggested or just use a counter to keep track of pushes that is happening currently. Submit task to drop the segments to persistExecutor only when the counter is zero, thus not relying on pushBarrier job submitted to pushExecutor to avoid interleaving.
Probably, new executor approach seems cleaner. I don't have preference of one over another.

[pjain1]

Added fix in #4815

[pjain1]

Fixed in #4815