[Format] Add a TensorMessage type

[asfimport]

Since all data message types at the moment are 1-dimensional, a "tensor" message will contain an array of dimensions and an order flag (C order vs. Fortran order) to enable data to be interpreted as multiple dimensions. This is similar to multidimensional arrays in APL or Fortran or MATLAB, ndarrays in NumPy, etc.

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

Related issues:

• [C++] Implement in-memory arrow::Tensor objects (is related to)
• [Python] Write and read tensors (with zero copy) into shared memory (is depended upon by)

_{Note: This issue was originally created as ARROW-550. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
@pcmoritz @robertnishihara could you elaborate on your needs in the Ray project? I want to make sure we're supporting your use case well.

[asfimport]

Philipp Moritz / @pcmoritz:
Hey Wes,

Thanks for the note, we are really excited about this feature!

On a high level, we need the ability to serialize arbitrary numpy arrays. This means:

1. Support for multiple dimensions (obviously)
2. Support for all numpy types
3. Support for numpy arrays that have more elements than can be indexed with an int32_t (we currently cannot do this but it is very important, we'd be interested in learning about your plans concerning this)

We have been using https://github.com/ray-project/ray/blob/master/src/numbuf/python/src/pynumbuf/adapters/numpy.cc and https://github.com/ray-project/ray/blob/master/src/numbuf/cpp/src/numbuf/dict.h so far which doesn't support 3) unfortunately.

4. Something that is not critical but might be good for performance is to have a way to specify the attributes of the numpy array (row-major, column-major, if it is transposed or not). That would allow us to avoid reordering/copying in some cases.

One problem we also have in this context (this is a little separate) is that we cannot currently use the arrow builder classes without an additional copy because they require buffers to be aligned to 64 bytes but the numpy arrays might not be. Finding a solution for this would also be great!

You might also want to look into the tensorflow tensors, see
https://github.com/tensorflow/tensorflow/blob/754048a0453a04a761e112ae5d99c149eb9910dd/tensorflow/core/framework/types.proto
https://github.com/tensorflow/tensorflow/blob/754048a0453a04a761e112ae5d99c149eb9910dd/tensorflow/core/framework/tensor.proto
They have some more types that make sense in the deep learning context.

– Philipp.

[asfimport]

Wes McKinney / @wesm:
I will look into the links you provided, thank you. Couple immediate follow up questions:

• Do you have any need of nested types with tensors?
• When you say "all NumPy types", do you mean all fixed-size data types excluding dtype=object arrays?

Having tensor metadata that supports more than int32_t size elements is not a problem – expanding normal Arrow arrays to exceed 2G would be a challenging proposition because Java does not support large buffers very well. I suspect that the tensor functionality would be limited to native code users.

As a first draft implementation in the Arrow C++ libraries I would add additional metadata for tensors and define tensor types that use the common metadata types defined in arrow::DataType.

[asfimport]

Philipp Moritz / @pcmoritz:
We do need tensors within other types but having only tensors of primitive types is fine. We have written our own sequence type to support nesting within lists or dicts, see https://github.com/ray-project/ray/blob/master/src/numbuf/cpp/src/numbuf/sequence.h. Something that comes up a lot for example in deep learning is dictionaries of tensors (these are weight collections for neural networks).

Yes, fixed size types are what we need. To handle dtype=object, we convert the tensors to lists and then use our sequence type.

Can you clarify what exactly you mean regarding the 2G limitation? If tensors larger than 2G are not accessible from Java that'd be ok for us for now.

Also using arrow::DataType sounds good!

[asfimport]

Philipp Moritz / @pcmoritz:
Would it be possible to use tensors inside an array (a) if the sizes of the individual tensors are larger than 2G or (b) if the sum of the sizes of the tensors is larger than 2G? Having (a) would be best and (b) is crucial.

[asfimport]

Wes McKinney / @wesm:
@pcmoritz we discussed this issue on the Arrow sync call yesterday. I think we should be able to propose a design for C++ users that will meet your needs (including accommodating arrays larger than 2G in the Arrow builder classes) – I will draw up a patch when I can and we can discuss.

[asfimport]

Philipp Moritz / @pcmoritz:
That's great and will help us a lot, thank you!

[asfimport]

Wes McKinney / @wesm:
PR: #435

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 435
#435