ARROW-13611: [C++] Scanning datasets does not enforce back pressure#11285

Closed

westonpace wants to merge 13 commits intoapache:masterfrom

westonpace:feature/ARROW-13611--scanning-datasets-backpressure

Commits on Oct 12, 2021

ARROW-13611: Added backpressure into the asynchronous scanner. Currently it only works for unordered scans. This commit adds backpressure in a way that bypasses the exec plan entirely and these u…
westonpace
committed
ARROW-13611: Lint
westonpace
committed
ARROW-13611: Missing some ARROW_EXPORT flags
westonpace
committed
ARROW-13611: Addressing PR comments. Fixed potential race condition where opening the backpressure while holding a lock. Added a unit test for the sink node
westonpace
committed
ARROW-13611: Clarified ordering behavior in comments
westonpace
committed
ARROW-13611: Cleaned up the backpressure factory functions a little for simplicity and making it more obvious what is happening. Added backpressure to the OrderBy sink node
westonpace
committed
ARROW-13611: Added ThreadPool::WaitForIdle and used it to make the backpressure tests more predictable. In addition, my previous attempt at fixing the race condition between the producer and the a…
westonpace
committed
ARROW-13611: General cleanup of the PR
westonpace
committed
ARROW-13611: Moved from ASSERT_TRUE to ASSERT_GE for more explicit assert
westonpace
committed
ARROW-13611: Enhanced the backpressure test a bit more. Ran lint/format. Removed disabled ordered backpressure test because I was just updating the same thing twice. It can be added back in when…
westonpace
committed
ARROW-13611: Fix memleak
westonpace
committed
ARROW-13611: Lint and another memleak fix. Turns out using PushGenerator as the source of an AsyncGenerator pipeline can be somewhat problematic. if the producer is not finished.
westonpace
committed
ARROW-13611: formatting
westonpace
committed