GH-33206: [C++] Add support for StructArray sorting and nested sort keys

[benibus]

Rationale for this change

We don't currently support sorting StructArrays despite already having the high-level facilities to do so. For instance, we allow passing multiple sort keys (based on FieldRefs) to sort record batches and tables - but the current implementations are fairly limited since nested refs aren't allowed (due to the burden of null flattening). Since #35197, we now have an easier way to do this.

What changes are included in this PR?

• Adds support for StructArray in sort_indices
• Adds support for nested sort keys in sort_indices for RecordBatch, ChunkedArray, and Table

Are these changes tested?

Yes (tests are included)

Are there any user-facing changes?

Yes

• Closes: [C++] Ensure sorting Table, RecordBatch can sort Struct and Dictionary cols #33206

[github-actions]

• Closes: [C++] Ensure sorting Table, RecordBatch can sort Struct and Dictionary cols #33206

[github-actions]

⚠️ GitHub issue #33206 has been automatically assigned in GitHub to PR creator.

[benibus]

Note that this doesn't currently extend nested key support to rank and selectk, but that would be a logical next step.

[benibus]

@pitrou Sorry, I pushed changes to this but forgot to re-ping...

There are a few things that could probably be tweaked still, but I wanted to get an opinion on the general approach here first (regarding sorting by entire struct fields).

[pitrou]

This looks more flexible now @benibus, thanks! Some comments below.

[pitrou]

Instead of leaking tmp_indices into the caller like this, can we have a toplevel AddField that doesn't take this parameter?

[benibus]

We could. Although doing it this way potentially avoids reallocating the vector for every path (it should retain its max capacity).

[pitrou]

I mean the toplevel AddField can delegate to the existing functions, but hide the tmp_indices dance from its caller.

[pitrou]

Oh, you mean accross all sort keys. Fair enough.

[pitrou]

That said, the fact that you're also passing an inout-parameter that gets appended to advocates for a struct carrying that state:

struct SortFieldPopulator {
 public:
  Result<std::vector<SortField>> FindSortKeys(...)

 protected:
  std::vector<SortField> sort_fields_;
  std::unordered_set<FieldPath> seen_;
  std::vector<int> tmp_indices_;
};

[pitrou]

Looking good, one minor suggestion.

[conbench-apache-arrow]

Conbench analyzed the 6 benchmark runs on commit 0f652201.

There were 26 benchmark results indicating a performance regression:

• Commit Run on ursa-thinkcentre-m75q at 2023-06-20 09:24:07Z
  • params=1048576/100/1/1, source=cpp-micro, suite=arrow-compute-vector-sort-benchmark
  • params=32768/16, source=cpp-micro, suite=parquet-column-io-benchmark
• and 24 more (see the report linked below)

The full Conbench report has more details.