Describe the bug
In non-preserve-order repartitioning mode, all input partition tasks share clones of the same SpillPoolWriter for each output partition. SpillPoolWriter used #[derive(Clone)] but its Drop implementation unconditionally set writer_dropped = true and finalized the current spill file. This meant that when the first input task finishes and its clone is dropped, the SpillPoolReader sees writer_dropped = true on an empty queue and returns EOF — silently discarding every batch subsequently written by the still-running input tasks.
This bug requires three conditions to trigger:
- Non-preserve-order repartitioning (so spill writers are cloned across input tasks)
- Memory pressure causing batches to spill to disk
- Input tasks finishing at different times (the common case with varying partition sizes)
To Reproduce
No response
Expected behavior
No response
Additional context
No response
Describe the bug
In non-preserve-order repartitioning mode, all input partition tasks share clones of the same
SpillPoolWriterfor each output partition.SpillPoolWriterused#[derive(Clone)]but itsDropimplementation unconditionally setwriter_dropped = trueand finalized the current spill file. This meant that when the first input task finishes and its clone is dropped, the SpillPoolReader seeswriter_dropped = trueon an empty queue and returns EOF — silently discarding every batch subsequently written by the still-running input tasks.This bug requires three conditions to trigger:
To Reproduce
No response
Expected behavior
No response
Additional context
No response