[Rust][DataFusion] Mixing of crossbeam channel and async tasks can lead to deadlock #27653
Description
[~edrevo] noticed, on #9523 (comment), that the use of crossbeam channels can potentially deadlock datafusion
The use of crossbeam channel is left over from earlier, non async implementations and get been fingered in some hangs that @seddonm1 has observed in DataFusion ). Specifically the crossbeam channel can block a thread when the channel is full or empty, which can result in blocking all the tokio executor threads and deadlocking the system
The proposal is is to use tokio's mpsc channels instead of crossbeam which can properly yield back to tokio to run another task when the channel is either full or empty. .
Reporter: Andrew Lamb / @alamb
Assignee: Andrew Lamb / @alamb
PRs and other links:
Note: This issue was originally created as ARROW-11802. Please see the migration documentation for further details.