pyarrow / pandas support for tensors (multi-dimensional arrays)

[pwais]

pyarrow appears to have a tensor type: https://arrow.apache.org/docs/python/generated/pyarrow.Tensor.html#pyarrow.Tensor

But numpy arrays do not get translated to Tensors when used in pandas dataframes:

if __name__ == '__main__':
  import numpy as np
  import pandas as pd
  a = np.array([
                [[1, 2],[1, 2],[1, 2]],
                [[1, 2],[1, 2],[1, 2]],
                [[1, 2],[1, 2],[1, 2]]])
  df = pd.DataFrame({'x':range(3), 'y':['a','b','b'], 'z':[a, a, a]})
  print(df)

  import pyarrow as pa
  import pyarrow.parquet as pq
  print(pa.__version__)

  table = pa.Table.from_pandas(df)
  pq.write_to_dataset(
        table,
        '/tmp/rows')

results in:

# python3 yay.py 
   x  y                                                  z
0  0  a  [[[1, 2], [1, 2], [1, 2]], [[1, 2], [1, 2], [1...
1  1  b  [[[1, 2], [1, 2], [1, 2]], [[1, 2], [1, 2], [1...
2  2  b  [[[1, 2], [1, 2], [1, 2]], [[1, 2], [1, 2], [1...
0.14.0
Traceback (most recent call last):
  File "yay.py", line 15, in <module>
    table = pa.Table.from_pandas(df)
  File "pyarrow/table.pxi", line 1174, in pyarrow.lib.Table.from_pandas
  File "/usr/local/lib/python3.6/dist-packages/pyarrow/pandas_compat.py", line 496, in dataframe_to_arrays
    for c, f in zip(columns_to_convert, convert_fields)]
  File "/usr/local/lib/python3.6/dist-packages/pyarrow/pandas_compat.py", line 496, in <listcomp>
    for c, f in zip(columns_to_convert, convert_fields)]
  File "/usr/local/lib/python3.6/dist-packages/pyarrow/pandas_compat.py", line 487, in convert_column
    raise e
  File "/usr/local/lib/python3.6/dist-packages/pyarrow/pandas_compat.py", line 481, in convert_column
    result = pa.array(col, type=type_, from_pandas=True, safe=safe)
  File "pyarrow/array.pxi", line 191, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 78, in pyarrow.lib._ndarray_to_array
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: ('Can only convert 1-dimensional array values', 'Conversion failed for column z with type object')

Might we be able to simply patch pandas_compat.py to do Tensor conversions?

[jorisvandenbossche]

There is no direct support in the arrow columnar format to store Tensors as column values.

They could be stored as lists of lists, or as binary blobs, see https://issues.apache.org/jira/browse/ARROW-5819 for an issue about adding an ExtensionType for that.

For a more detailed answer, see this recent thread on the mailing list: https://lists.apache.org/thread.html/9b142c1709aa37dc35f1ce8db4e1ced94fcc4cdd96cc72b5772b373b@%3Cdev.arrow.apache.org%3E

[pwais]

@jorisvandenbossche thanks for the info and the link to @wesm 's reply. When you say "arrow columnar format", do you really mean the parquet columnar format?

Hmm, it looks like Tensors are IPC-only?

arrow/python/pyarrow/tests/test_tensor.py

Line 1 in 74b9294

# Licensed to the Apache Software Foundation (ASF) under one

Why is that? One of those tickets is from 2017, so I'm a little confused why there's this disparity. Perhaps pyarrow.Tensor was designed early in the project and there was some expectation that there would be 1:1 correspondence between arrow types and parquet types?

[jorisvandenbossche]

When you say "arrow columnar format", do you really mean the parquet columnar format?

No, I actually mean the arrow memory format. The standardized, language independent in-memory column format is a corner piece of the project. And it is here that Tensors are not supported as "scalars" (single values) in a column. But, I think that the same limitation applies to parquet as well.
(See also this FAQ about arrow vs parquet: https://github.com/apache/arrow/blob/master/site/faq.md#what-is-the-difference-between-apache-arrow-and-apache-parquet, will be online after the next release)

Hmm, it looks like Tensors are IPC-only?

Tensors are indeed supported in the IPC message format: https://arrow.apache.org/docs/format/IPC.html#encapsulated-message-format
In addition, they are also supported in the serialization functionalities of pyarrow, but I think this is more specific for the python implementation and not a general Arrow feature.

One of those tickets is from 2017, so I'm a little confused why there's this disparity

It seems that storing Tensors in columns is not the highest priority of the current people involved. But contributions for the open issues are certainly very welcome!

In general, the Tensor classes are mostly independent from the columnar data structures.

[jorisvandenbossche]

I didn't directly find an issue for the idea of converting arrays to nested lists, so opened: https://issues.apache.org/jira/browse/ARROW-5857

[pwais]

Ah, thanks for the FAQ, that's super helpful!

[wesm]

Why is that? One of those tickets is from 2017, so I'm a little confused why there's this disparity. Perhaps pyarrow.Tensor was designed early in the project and there was some expectation that there would be 1:1 correspondence between arrow types and parquet types?

Tensors aren't part of the Arrow columnar format. Let's continue the discussion on the mailing list or JIRA

[vpipkt]

ARROW-5857 was marked dupe of ARROW-5645

[NightMachinery]

I have a (10**6, 10**4) numpy array. I want to store this in an efficient format and be able to load batches from this file to use in an out-of-core algorithm. Is Arrow suitable for this purpose? How do I do it?

PS: I actually generate the big numpy array as 10 parts of (10**5, 10**4); so I need to be able to append to the storage file.

[jorisvandenbossche]

In general, Arrow will be able to handle such a table, but it is certainly not the most optimal format for this kind of data (especially since we don't yet have good support of converting such an array to a single (nested) column zero-copy, which is what the above discussion is about).

But if you want to discuss this, please open an new issue or start a mailing thread on the user mailing list, instead of using this old / closed issue.

[rabernat]

I found this issue while investigating how we might round-trip Xarray datasets (which contain many ndarray) through Arrow.

My impression from the recent comments it that Arrow is really not meant for this. However, given the strong growth of Arrow, many people are nevertheless trying to put ndarrays into it. Huggingface datasets, for example, uses an ExtensionArray to store ndarrays in arrow. Would you say that is aligned with best practices?

@NightMachinary and others may want to consider using Zarr as an efficient storage format for nd-arrays.