GH-41183: [C++][Python] Expose recursive flatten for lists on list_flatten kernel function and pyarrow bindings

[ZhangHuiGui]

Rationale for this change

Expose recursive flatten for logical lists on list_flatten kernel function and pyarrow bindings.

What changes are included in this PR?

1. Expose recursive flatten for logical lists on list_flatten kernel function
2. Support [Large]ListView for some kernel functions: list_flatten,list_value_length, list_element
3. Support recursive flatten for pyarrow bindinds and simplify [Large]ListView's pyarrow bindings
4. Refactor vector_nested_test.cc for better support [Large]ListView types.

Are these changes tested?

Yes

Are there any user-facing changes?

Yes.

1. Some kernel functions like: list_flatten, list_value_length, list_element would support [Large]ListView types
2. list_flatten and related pyarrow bindings could support flatten recursively with an ListFlattenOptions.

• GitHub Issue: [C++][Python] Expose recursive flatten for lists on list_flatten kernel function and pyarrow bindings #41183

[github-actions]

⚠️ GitHub issue #41183 has been automatically assigned in GitHub to PR creator.

[ZhangHuiGui]

@felipecrv @mapleFU PTAL this if you're interested!

[felipecrv]

You should add a bool recursive parameter (without default value) and skip the for loop if it's false to match existing usage.

[ZhangHuiGui]

The logic here seems correct for the recursive = false condition except one more value_kind check on is_list(value_kind) || is_list_view(value_kind).

And it's the output resolver(a callback to identify the kernel functions's output type) which called only once in expression bind or CallFunction's, and have little impact on kernel path. So maybe the logic here could be retained?

[felipecrv]

Maybe. I started another thread and asked for clarification #41295 (comment)

[felipecrv]

@ZhangHuiGui if you're dealing with bugs/failures in tests, it might be because of this
#41295 (comment)

[felipecrv]

Before merge I will do a more detailed review (I don't think I will find issues).

@jorisvandenbossche what you think about the Python changes?

[ZhangHuiGui]

Pyarrow's mainly changes should be here. [Large]ListViewArray does not seem to need to maintain an independent flatten interface separately, and can directly use the one provided by BaseListArray.

[felipecrv]

Please accept the patch suggestions containing typos and grammar corrections.

After these changes, I will get a Python person to give a quick look at the Python parts, then I will merge.

[danepitkin]

LGTM (from python perspective)

[paleolimbot]

Thanks!

I took a look through the Python changes, which seem reasonable except for possibly an explicit test to make sure the value of recursive makes it all the way to C++.

I don't have much experience with the class hierarchy of Arrays and so I can't speak to that change (but it seems like a good idea to only maintain one version of that documentation/interface).

[paleolimbot]

Should there also be a test here where flatten(True) and flatten(False) return different things?

[ZhangHuiGui]

Yes, the recursive parameter defaults to False in the flatten interface, but some of the previous test cases do not include this test. We should add such a test case here.

[paleolimbot]

I see: assert arr2.flatten(True).equals(arr0) is the case that I had in mind and it was already added (I just missed the distinction between arr1 and arr0). The case you added helps make that distinction clearer...thank you!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 5e986be.

There was 1 benchmark result with an error:

• Commit Run on arm64-m6g-linux-compute at 2024-05-01 01:15:07Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-15, scale_factor=10

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 11 possible false positives for unstable benchmarks that are known to sometimes produce them.