-
Notifications
You must be signed in to change notification settings - Fork 4k
GH-32190: [C++][Compute] Implement cumulative prod, max and min functions #36020
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
|
bkietz
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for doing this!
This looks good, a few minor comments:
python/pyarrow/compute.py
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jorisvandenbossche Does this need an alias?
| CumulativeOptions, | |
| CumulativeOptions, | |
| CumulativeOptions as CumulativeSumOptions, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For backwards compatibility, that's probably useful, yes (also on the C++ side this was done).
(although we should maybe actually add it as its own class and deprecate it, so we can remove the alias at some point. But this can also be done as a follow-up later)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Follow up issue here #36240
f4de821 to
4bd7ca2
Compare
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
| template <typename Value> | ||
| static constexpr Value value{std::numeric_limits<Value>::min()}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These cumulative ops aren't currently instantiated for decimal types anyway: https://github.com/apache/arrow/blob/44edc27/cpp/src/arrow/compute/kernels/codegen_internal.h#L1070-L1100
... if you feel like filing a follow up issue for them, it should be fairly straightforward to specify these as specializations:
| template <typename Value> | |
| static constexpr Value value{std::numeric_limits<Value>::min()}; | |
| template <typename Value> | |
| static constexpr Value value{std::numeric_limits<Value>::min()}; | |
| template <> | |
| static constexpr Decimal128 value<Decimal128> = Decimal128::GetMinSentinel(); |
Multiplication will require more effort since we can't construct 1 without knowing the scale.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll remove them for this pr. Will send another pr for decimals.
bkietz
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks!
|
CI failure is a backpressure test flake-out in AsOfJoin, unrelated. |
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
|
Conbench analyzed the 6 benchmark runs on commit There were 6 benchmark results indicating a performance regression:
The full Conbench report has more details. |
Rationale for this change
Implement cumulative prod, max and min compute functions
What changes are included in this PR?
CumulativeSumOptionstoCumulativeOptionsfor reusability.GenericFromScalar(GenericToScalar(std::nullopt)) != std::nullopt.NaN.I'll explain some of the changes in comments.
Are these changes tested?
Yes, in vector_accumulative_ops_test.cc and test_compute.py
Are there any user-facing changes?
No. The data members of
CumulativeSumOptionsare changed, but the member functions behave as before. And std::optional also can be constructed directly from T. So users should not feel any difference.