ARROW-11725: [Rust][DataFusion] Make use of the new divide_scalar kernel in arrow #9543
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
abreis
commented
Feb 21, 2021
| macro_rules! binary_string_array_op_scalar { | ||
| ($LEFT:expr, $RIGHT:expr, $OP:ident) => {{ | ||
| let result = match $LEFT.data_type() { | ||
| let result: Result<Arc<dyn Array>> = match $LEFT.data_type() { |
Contributor
Author
There was a problem hiding this comment.
I could not find a way to do this type erasure directly in BinaryExpr::evaluate (L419, which only had scalar operations on BooleanArray before), so I'm doing it here and also for binary_array_op_scalar (L244).
alamb
approved these changes
Feb 22, 2021
Contributor
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
Thanks @abreis -- looks really nice to me. @jorgecarleitao or @Dandandan any thoughts?
alamb
approved these changes
Feb 23, 2021
Contributor
|
I plan to merge this once the CI goes green |
Contributor
Author
|
CI failure seems unrelated. Note that the first commit already passed CI, and this second commit only changes a few error strings, so it should be safe to merge. |
Contributor
|
The integration failure looks like https://issues.apache.org/jira/browse/ARROW-11717 |
Contributor
|
Merged. 🎉 Thanks @abreis |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a small PR to make DataFusion use the just-merged
divide_scalararrow kernel (#9454).Performance-wise:
arrowside, this specialized kernel is ~40-50% faster than the standarddivide, mostly due to not having to check for divide-by-zero on every row;datafusionside, it can now skip thescalar.to_array_of_size(num_rows)allocation, which should be a decent win for operations on large arrays.The eventual goal is to have
op_scalarvariants for every arithmetic operation —dividewill show the biggest performance gains but all variants should save DataFusion a (possibly expensive) allocation.