ARROW-16419: [Python] Properly wait for ExecPlan to finish #13035
Conversation
| cdef extern from "arrow/util/future.h" namespace "arrow" nogil: | ||
| cdef cppclass CFuture_Void" arrow::Future<>": | ||
| CStatus status() |
There was a problem hiding this comment.
Surprising we've made it this far without exposing Future. I wonder if we want to add a WaitForFinish method on ExecPlan instead of exposing Future (essentially a sync mirror-api) like we do in the Scanner.
I don't really have any problems with this approach though. Just making an observation.
There was a problem hiding this comment.
I suppose at some point we'll want to expose async APIs in Python, so we're probably bound to interface with Future in Cython anyway.
|
How severe this issue is? Does this belong to a new feature? I'm not sure whether we should block the release on it or not. cc @pitrou |
|
This by itself is not blocking but see #13036 which is the fix for the segfault I was running into on the ML. |
This builds on top of #13035 which is also important for avoiding segmentation faults. On top of that there were a few more problems: * The python was using `SourceNodeOptions::FromTable` which is a rather dangerous method (mainly useful for unit testing) as it doesn't share ownership of the input table (even worse, it takes a const ref). Python was not keeping the table alive and it was maybe possible for the table to deleted out from under the plan (I'm not entirely sure this was causing issues but it seemed risky). I switched to TableSourceNode which shares ownership of the table (and is a bit more efficient). * Setting use_threads to False did nothing because `_perform_join` was not passing the arg on to `execplan`. * When fixing the above and running with `use_threads=False` it was creating a single thread executor but the current best practice is to pass in nullptr. * Finally, the actual bug was my improper fix in #12894 . I had still left a small window open for `End` to be called between `Submit` and `AddTask` which would allow the task to be submitted but not participate in setting `finished` on the node. Closes #13036 from westonpace/bugfix/ARROW-16417--segfault-in-python-join Lead-authored-by: Weston Pace <weston.pace@gmail.com> Co-authored-by: David Li <li.davidm96@gmail.com> Signed-off-by: David Li <li.davidm96@gmail.com>
|
Benchmark runs are scheduled for baseline = 22a1885 and contender = 3f2c33f. 3f2c33f is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
5 participants
No description provided.