[Python] Fixed size lists of numeric types without nulls could be converted to numpy with zero-copy

[spenczar]

Describe the enhancement requested

Today, FixedSizeListArrays seem to demand that zero_copy_only be set to False unnecessarily.

For example:

Create a fixed size list array: 3 rows, with fixed-size of 2:

import pyarrow as pa
data = pa.FixedSizeListArray.from_arrays([1, 2, 3, 4, 5, 6], 2)
print(data)

<pyarrow.lib.FixedSizeListArray object at 0x104abe140>
[
  [
    1,
    2
  ],
  [
    3,
    4
  ],
  [
    5,
    6
  ]
]

Calling to_numpy() throws an error:

data.to_numpy()

ArrowInvalid: Needed to copy 1 chunks with 0 nulls, but zero_copy_only was True

But if I work with buffers directly, I can easily get it to work:

nparray = np.frombuffer(data.buffers()[2], dtype=np.int64())
print(nparray)

array([1, 2, 3, 4, 5, 6])

We also know enough to even give it the right ndarray shape:

nparray_shaped = np.frombuffer(data.buffers()[2], dtype=np.int64()).reshape(len(data), data.type.list_size)
print(nparray_shaped)

array([[1, 2],
       [3, 4],
       [5, 6]])

---

I propose that FixedSizeListArray.to_numpy() should return numpy arrays with zero copy if the FixedSizeList's type is an integer or floating point type, since those are safe to convert, and if no nulls are present.

I also propose that it reshape the output to be an ndarray which matches the FixedSizeList's shape.

Component(s)

Python

[jorisvandenbossche]

The problem is that to_numpy() for a fixed size list array doesn't give you this flat (or nd) array of the values, but an object dtype array of sub-arrays:

>>> data.to_numpy(zero_copy_only=False)
array([array([1, 2]), array([3, 4]), array([5, 6])], dtype=object)

So while it is true that in case of numeric type without missing values, the underlying values are being converted zero copy, and the sub-arrays in the arrays above are zero-copy slices of this converted array, but the actual object-dtype numpy array that is returned in the snippet above is still a newly allocated array.
So it is a bit ambiguous here what "zero copy" would mean exactly.

But if I work with buffers directly, I can easily get it to work:

Sidenote, there is actually another API to directly get this numpy array, without having to go through the buffers manually:

# the underlying values as a pyarrow array (only beware that this could contain "garbage" values in case of nulls)
>>> data.values
<pyarrow.lib.Int64Array object at 0x7f41b7682aa0>
[
  1,
  2,
  3,
  4,
  5,
  6
]
# in case of no missing values, this can be converted zero-copy to numpy
>>> data.values.to_numpy()
array([1, 2, 3, 4, 5, 6])

[spenczar]

The problem is that to_numpy() for a fixed size list array doesn't give you this flat (or nd) array of the values, but an object dtype array of sub-arrays.

True, but is this desirable? I can't think of a scenario where that is a useful return shape. In practice, my group currently always reshapes that into a 2D array.

FixedShapeTensorArray has a to_numpy_ndarrray method which addresses this usage issue:

arrow/python/pyarrow/array.pxi

Lines 3149 to 3161 in f45a9e5

	def to_numpy_ndarray(self):
	"""
	Convert fixed shape tensor extension array to a numpy array (with dim+1).
	Note: ``permutation`` should be trivial (``None`` or ``[0, 1, ..., len(shape)-1]``).
	"""
	if self.type.permutation is None or self.type.permutation == list(range(len(self.type.shape))):
	np_flat = np.asarray(self.storage.values)
	numpy_tensor = np_flat.reshape((len(self),) + tuple(self.type.shape))
	return numpy_tensor
	else:
	raise ValueError(
	'Only non-permuted tensors can be converted to numpy tensors.')

Maybe a solution is for FixedSizeListArray to have a to_numpy_ndarray method? It'd be a somewhat crummy API, since you'd almost never want to use to_numpy, but it'd preserve the current behavior.

Sidenote, there is actually another API to directly get this numpy array, without having to go through the buffers manually:

Very useful to know, thanks! I'll add an issue that those should be documented for the ListArray, LargeListArray, and FixedSizeListArray types (see https://arrow.apache.org/docs/python/generated/pyarrow.FixedSizeListArray.html#pyarrow.FixedSizeListArray.values).

[jorisvandenbossche]

The problem is that to_numpy() for a fixed size list array doesn't give you this flat (or nd) array of the values, but an object dtype array of sub-arrays.

True, but is this desirable? I can't think of a scenario where that is a useful return shape. In practice, my group currently always reshapes that into a 2D array.

I agree it is often not very useful, but I think it is nonetheless the sensible behaviour for to_numpy(): for any other array type, to_numpy() always gives you a numpy representation of the pyarrow array, i.e. a 1D array of the same length.
I think that is a clear contract for to_numpy(), and if we keep those restrictions for list arrays as well (for consistency accross array types), I don't think there is any other option to return in to_numpy() than the current (not super useful) object array.

For example, one case where this is useful is when converting to pandas, where this object dtype array is currently the only option given pandas has not yet real support for nested data (apart from their experimental support for using pyarrow directly).

And that is also the reason we used another name for FixedShapeTensorArray to get the nd array. We could indeed add this to FixedSizeListArray as well.

[spenczar]

to_numpy() always gives you a numpy representation of the pyarrow array, i.e. a 1D array of the same length.

Could you say more about what you mean here? A numpy ndarray of shape (5, 2) has length 5:

import numpy as np
vals = np.ones((5, 2))
print(vals)
# [[1. 1.]
# [1. 1.]
# [1. 1.]
# [1. 1.]
# [1. 1.]]
print(len(vals))
# 5

Likewise, you can do vals[i] for i in [0, 4]; vals[6] will give you an IndexError, etc.

Perhaps you mean something different by "length" though?

one case where this is useful is when converting to pandas

Isn't that the job of to_pandas(), not to_numpy()?

[jorisvandenbossche]

to_numpy() always gives you a numpy representation of the pyarrow array, i.e. a 1D array of the same length.

Could you say more about what you mean here? A numpy ndarray of shape (5, 2) has length 5:

It has length 5 indeed, but it is not a 1D array (and Arrow only has logical 1D arrays)
But it's true that for many use cases, this 2D array will behave correctly as we would expect for a 1D array (like iterating / indexing will do that on the first axis, which is the axis with the correct length)

[spenczar]

Right, thanks for the clarification.

How do you suggest we proceed? It seems like adding a to_numpy_ndarray method to the FixedSizeList type would be safe and preserve the behavior of to_numpy, at the cost of some user confusion. I don't know enough about the overall philosophy of Arrow to say whether that's the right tradeoff, but it might be.