[C++/Python] Add pandas-like take method to Array

[asfimport]

We should add a take method to Array/ChunkedArray/Column that takes a list of indices and returns a reordered array.

For reference, see Pandas' interface: https://github.com/pandas-dev/pandas/blob/2cbdd9a2cd19501c98582490e35c5402ae6de941/pandas/core/arrays/base.py#L466

Reporter: Uwe Korn / @xhochy

Related issues:

• [C++] Implement take kernel functions (is blocked by)
• [Python] fancy indexing on pa.array (duplicates)
• [C++] Implement Take on ChunkedArray for DataFrame use (relates to)

_{Note: This issue was originally created as ARROW-2667. Please see the migration documentation for further details.}

[asfimport]

Tom Augspurger / @TomAugspurger:
Note that pandas' take is a bit complicated by trying to satisfy two APIs simultaneously.

 

There's the NumPy-style take from https://docs.scipy.org/doc/numpy/reference/generated/numpy.take.html, where negative indices mean slices from the end.

 

And then there's the "pandas" style take where -1 means "indicator for missing values, which should be filled with the na_value parameter." Other negative numbers are not allowed.

 

I'm not sure which is more appropriate for Arrow, but wanted to share a bit of background.

[asfimport]

Uwe Korn / @xhochy:
My goal is to have something that I can use to fulfill the Pandas API. A first version in Arrow could also fully omit the treatment of negative values and simply work on non-negative integers alone.

As we have the valid bitmap for marking missing values, I would lean more towards the numpy way of things.

[asfimport]

Dhruv Madeka / @dmadeka:
Would this be the right issue for adding a slice function to Table? It seems to be useful in writing multiple files

[asfimport]

Wes McKinney / @wesm:
No, this issue is about random access selection and not slicing

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:

Note that pandas' take is a bit complicated by trying to satisfy two APIs simultaneously.
[...]
And then there's the "pandas" style take where -1 means "indicator for missing values, which should be filled with the na_value parameter." Other negative numbers are not allowed.

I think this distinction in Arrow is less relevant / easier to deal with. The indices can also have nulls, resulting of nulls in the returned array. This basically takes the role of the -1 in the "pandas style" take.

(working on the Array part in ARROW-5291 / #4281)

[asfimport]

Neal Richardson / @nealrichardson:
@jorisvandenbossche is this issue still valid or did your work on 5291 resolve it?

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
I only added it to Array, not yet to ChunkedArray, Column or Table, since take is not yet implemented for ChunkedArray on the C++ side. So it's not yet fully resolved.

[asfimport]

Wes McKinney / @wesm:
I would suggest that work on a ChunkedArray take be put off to some later time since it's a fairly complex project (because the "frame of reference" changes from chunk to chunk so the indices have to be adjusted, not simple with the current Take implementation)

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
@wesm you renamed this issue to only be about Array (and opened ARROW-5454 for the ChunkedArray part). So then this can be closed? (the python Array part was tackled in ARROW-5291)

Edit: the other issue is only about C++, so we can keep this open for the Python side of course.

[asfimport]

Wes McKinney / @wesm:
Yes, going ahead and closing