[C++][Dataset] Backpressure broken on ordered scans #29777
Description
ARROW-13611 adds a backpressure mechanism that works for unordered scans. However, this backpressure is not properly applied on ordered (i.e. ScanBatches and not ScanBatchedUnordered) scans.
The fix will be to modify the merge generator used on ordered scans so that, while it still will read ahead somewhat on several files, it will never deliver batches except from the currently read file.
Reporter: Weston Pace / @westonpace
Assignee: Weston Pace / @westonpace
PRs and other links:
Note: This issue was originally created as ARROW-14192. Please see the migration documentation for further details.