ARROW-13882: [C++] Improve min_max/hash_min_max type support #11111

lidavidm · 2021-09-08T19:23:22Z

This adds support for non-nested types to both, except for MonthDayNanoInterval (which is not really sortable). hash_min_max additionally lacks binary/string-like types as they require a different approach (I will file a followup).

github-actions · 2021-09-08T19:23:42Z

https://issues.apache.org/jira/browse/ARROW-13882

lidavidm · 2021-09-13T11:57:50Z

In light of the proposal to clarify how DayTime intervals are handled some changes here should probably be reverted.

lidavidm · 2021-09-13T12:25:08Z

(Removed support for DayTimeInterval since it's only partially orderable.)

pitrou

Very nice, thank you. Just a couple questions and nits.

pitrou · 2021-09-13T16:55:33Z

cpp/src/arrow/array/validate.cc

@@ -498,6 +498,11 @@ struct ValidateArrayFullImpl {

  Status Visit(const StructType& type) {
    // Validate children
+    if (data.child_data.size() != static_cast<size_t>(type.num_fields())) {


I'm a bit surprised. This should already be checked by the ValidateArray top-level function, no?

Hmm. Odd, I do remember getting a crash for this reason here. But there is indeed a check:

arrow/cpp/src/arrow/array/validate.cc

Lines 367 to 374 in 87b2fcd

if (type.id() != Type::EXTENSION) {

if (data.child_data.size() != static_cast<size_t>(type.num_fields())) {

return Status::Invalid("Expected ", type.num_fields(),

" child arrays in array "

"of type ",

type.ToString(), ", got ", data.child_data.size());

}

}

pitrou · 2021-09-13T17:00:32Z

cpp/src/arrow/compute/kernels/aggregate_basic_internal.h

  T min = true;
  T max = false;
  bool has_nulls = false;
  bool has_values = false;
 };

 template <typename ArrowType, SimdLevel::type SimdLevel>
-struct MinMaxState<ArrowType, SimdLevel, enable_if_integer<ArrowType>> {
+struct MinMaxState<ArrowType, SimdLevel, enable_if_physical_integer<ArrowType>> {


It seems this will generate different template specializations for types with the same underlying c_type. Is it desirable?
(note that GetValues is called with the output type independently from the ArrowType parameter)

I changed this and the one below to use PhysicalType. I also replaced GetValues with MakeScalar (since now the static type doesn't match the runtime type).

pitrou · 2021-09-13T17:03:13Z

cpp/src/arrow/compute/kernels/aggregate_basic_internal.h

+      if (value < util::string_view(this->min)) {
+        this->min = std::string(value);
+      }
+      if (value > util::string_view(this->max)) {


Should this be else if? Presumably value cannot be bother smaller than the min and greater than the max, and it would spare a string comparison.

cpp/src/arrow/compute/kernels/aggregate_basic_internal.h

pitrou · 2021-09-13T17:06:15Z

cpp/src/arrow/compute/kernels/aggregate_basic_internal.h

-  Status Visit(const DataType&) {
-    return Status::NotImplemented("No min/max implemented");
+  Status Visit(const DataType& ty) {
+    return Status::NotImplemented("No min/max implemented for ", ty);
  }

  Status Visit(const HalfFloatType&) {
    return Status::NotImplemented("No min/max implemented");


Update message here as well?

pitrou · 2021-09-13T17:21:36Z

cpp/src/arrow/compute/kernels/hash_aggregate.cc

    kernel = MakeKernel(std::move(argument_type), MinMaxInit<T>);
    return Status::OK();
  }

+  // MSVC2015 apparently doesn't compile this properly if we use
+  // enable_if_floating_point


pitrou · 2021-09-13T17:22:24Z

cpp/src/arrow/compute/kernels/hash_aggregate.cc

@@ -1866,17 +1940,40 @@ Result<std::unique_ptr<KernelState>> MinMaxInit(KernelContext* ctx,

 struct GroupedMinMaxFactory {
  template <typename T>
-  enable_if_number<T, Status> Visit(const T&) {
+  enable_if_physical_integer<T, Status> Visit(const T&) {


Same remark as for the scalar aggregations: this will generate a separate kernel per logical type?

pitrou · 2021-09-13T17:23:14Z

cpp/src/arrow/compute/kernels/hash_aggregate_test.cc

-    [0.75,  null],
-    [null,  3]
-                        ])"});
+      // bool, day time interval


Hmm, I don't see any interval in the type definition above. Is this comment outdated?

I fixed the comments/docs.

pitrou · 2021-09-13T17:25:34Z

docs/source/cpp/compute.rst

@@ -226,6 +229,9 @@ Notes:

 * \(3) Output is a ``{"min": input type, "max": input type}`` Struct.

+  The month-day-nano interval type is not supported as it cannot be
+  sorted.


Neither is day-time, right?

pitrou · 2021-09-13T17:25:53Z

docs/source/cpp/compute.rst

-* \(3) Output is a ``{"min": input type, "max": input type}`` Struct scalar.
+* \(3) Output is a ``{"min": input type, "max": input type}`` Struct array.
+
+  The month-day-nano interval type is not supported as it cannot be


Same question.

pitrou

+1, thank you @lidavidm

This adds support for non-nested types to both, except for MonthDayNanoInterval (which is not really sortable). hash_min_max additionally lacks binary/string-like types as they require a different approach (I will file a followup). Closes apache#11111 from lidavidm/arrow-13882 Lead-authored-by: David Li <li.davidm96@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>

lidavidm changed the title ~~ARROW-13882: [C++] Improve min_max/hash_min_max type support.~~ ARROW-13882: [C++] Improve min_max/hash_min_max type support Sep 8, 2021

github-actions bot added the Component: C++ label Sep 8, 2021

lidavidm force-pushed the arrow-13882 branch from 234913b to f49f7cf Compare September 13, 2021 12:24

pitrou reviewed Sep 13, 2021

View reviewed changes

lidavidm force-pushed the arrow-13882 branch 2 times, most recently from e9087e0 to c05cb96 Compare September 15, 2021 15:22

lidavidm and others added 5 commits September 16, 2021 15:05

ARROW-13882: [C++] Support more types in min_max

46d9b98

ARROW-13882: [C++] Fix MSVC

4dbcf4f

ARROW-13882: [C++] Fix MSVC

cb1bd17

ARROW-13882: [C++] Remove AllTimeUnits

b9f680c

Use anonymous namespace

51d1386

pitrou force-pushed the arrow-13882 branch from c05cb96 to 51d1386 Compare September 16, 2021 13:08

pitrou approved these changes Sep 16, 2021

View reviewed changes

pitrou closed this in 2c405da Sep 16, 2021

asfimport mentioned this pull request Apr 20, 2022

[C++] Add compute function min_max support for more types #29499

Closed

	if (type.id() != Type::EXTENSION) {
	if (data.child_data.size() != static_cast<size_t>(type.num_fields())) {
	return Status::Invalid("Expected ", type.num_fields(),
	" child arrays in array "
	"of type ",
	type.ToString(), ", got ", data.child_data.size());
	}
	}

ARROW-13882: [C++] Improve min_max/hash_min_max type support #11111

ARROW-13882: [C++] Improve min_max/hash_min_max type support #11111

Uh oh!

Conversation

lidavidm commented Sep 8, 2021

Uh oh!

github-actions bot commented Sep 8, 2021

Uh oh!

lidavidm commented Sep 13, 2021

Uh oh!

lidavidm commented Sep 13, 2021

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants