[C++][Acero] Acero's shared (per-thread) temp vector stack usage may overflow

[zanmato1984]

Describe the enhancement requested

Currently the size of Acero's temp stack is hardcoded in source:

arrow/cpp/src/arrow/acero/query_context.cc

Line 56 in 25bb627

memory_pool(), 32 * util::MiniBatch::kMiniBatchLength * sizeof(uint64_t)));

We've observed issues of hang/crash caused by this stack overflow, to name two: #39582, #39951 . And the hardcoded stack size has been once bumped, along with a more explicit stack overflow error, to solve them: #40007. But as long as the stack size is still hardcoded (and not insanely big), it won't be surprising if the overflow happens again for a more complex plan. And once that happened, one wouldn't be able to mitigate this issue without patching the source code (with a bigger stack size) and building the binary from scratch.

I think an env var might be well suitable to tune this stack size at runtime, to help the user quick mitigate the overflow.

But if we get lucky, overflow may still happen - a concrete case: https://github.com/apache/arrow/pull/41335/files#diff-6c3d49ae137b8279afb2610f6d16010618e451b4da2b4279637cf44175b0cd71R3206-R3254

Component(s)

C++

[pitrou]

Issue resolved by pull request 41335
#41335