[Python] Convert SerializedPyObject to/from sequence of component buffers with minimal memory allocation / copying #17781
Description
See discussion on Dask org:
It would be valuable for downstream users to compute the serialized payload as a sequence of memoryview-compatible objects without having to allocate new memory on write. This means that the component tensor messages must have their metadata and bodies in separate buffers. This will require a bit of work internally reassemble the object from a collection of pyarrow.Buffer objects
see also ARROW-1509
Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm
Related issues:
- [Python] Write serialized object as a stream of encapsulated IPC messages (relates to)
- [Python] Read and write pandas.DataFrame in pyarrow.serialize by decomposing the BlockManager rather than coercing to Arrow format (is related to)
PRs and other links:
Note: This issue was originally created as ARROW-1783. Please see the migration documentation for further details.