[Python] Read and write pandas.DataFrame in pyarrow.serialize by decomposing the BlockManager rather than coercing to Arrow format

[asfimport]

See discussion in dask/distributed#931

This will permit zero-copy reads for DataFrames not containing Python objects. In the event of an ObjectBlock these arrays will not be worse than pickle to reconstruct on the receiving side

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

Related issues:

• [Python] Convert SerializedPyObject to/from sequence of component buffers with minimal memory allocation / copying (relates to)
• [Python] Improve performance of serializing object dtype ndarrays (relates to)
• [Python] More efficient serialization of pandas Index types in custom serialization from ARROW-1784 (is related to)

PRs and other links:

• GitHub Pull Request #1390

_{Note: This issue was originally created as ARROW-1784. Please see the migration documentation for further details.}

[asfimport]

Matthew Rocklin / @mrocklin:
Note that matching the BlockManager itself is not important for us. I could imagine doing things columnwise as well if that feels cleaner semantically or more future-proof.

[asfimport]

Wes McKinney / @wesm:
It's hard to prevent a memory doubling on receipt if you go column-wise (e.g. pd.DataFrame(data) where data is a dict of columns will double memory). So I think as long as we avoid memory doubling we are good

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 1390
#1390