[Python] Table.from_pandas does not create chunked_arrays.

[asfimport]

When creating a large enough array, pyarrow raises an exception:

import numpy as np
import pandas as pd
import pyarrow as pa

x = list('1' * 2**31)
y = pd.DataFrame({'x': x})
t = pa.Table.from_pandas(y)
# ArrowInvalid: BinaryArrow cannot contain more than 2147483646 bytes, have 2147483647

The array should be chunked for the user. As is, data frames with >2 GiB in binary data will struggle to get into arrow.

Reporter: Left Screen
Assignee: Wes McKinney / @wesm

Related issues:

• [C++] Parquet arrow::Table reads error when overflowing capacity of BinaryArray (is related to)

Externally tracked issue: #1673

PRs and other links:

• GitHub Pull Request #1740

_{Note: This issue was originally created as ARROW-2227. Please see the migration documentation for further details.}

[asfimport]

Left Screen:
https://issues.apache.org/jira/browse/ARROW-1167 appears to have added support for this, but something is still overflowing.

[asfimport]

Wes McKinney / @wesm:
Thanks, marked for 0.9.0

[asfimport]

Left Screen:
It looks like parquet.ParquetFile.read() doesn't appropriately combine row groups for binary data either and you get the same exception.  If you read each row group and then concat the tables, you're fine though.

 

[asfimport]

Wes McKinney / @wesm:
That may be a separate bug, we should investigate

[asfimport]

Antoine Pitrou / @pitrou:
The snippet produces a core dump here (**). I think it is related to code touched by ARROW-2141 and especially the comment I posted here about an incorrect cast to 32-bit: https://github.com/apache/arrow/pull/1689/files#r173777819 .

(as a sidenote, I don't know why ARROW-2141 is necessary to allow conversion from Numpy but conversion from Pandas is already implemented. I suspect different paths are taken?)

(re-sidenote, where does the 2GB limit stem from? the desire to have shorter offset arrays?)

I would recommend deferring this to 0.10.0 so that we can sanitize the whole situation. There seem to be separate code paths converting Python bytes objects to Arrow data, with slightly different strategies...

(**) gdb backtrace:

#0  __memcpy_avx_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S:245
#1  0x00007fffc23dc5e8 in arrow::BufferBuilder::UnsafeAppend (this=0x7fffffffb538, data=0x7fff3f01d030, length=-2147483648)
    at /home/antoine/arrow/cpp/src/arrow/buffer.h:285
#2  0x00007fffc23dc236 in arrow::BufferBuilder::Append (this=0x7fffffffb538, data=0x7fff3f01d030, length=-2147483648)
    at /home/antoine/arrow/cpp/src/arrow/buffer.h:255
#3  0x00007fffc242b30f in arrow::TypedBufferBuilder<unsigned char>::Append (this=0x7fffffffb538, 
    arithmetic_values=0x7fff3f01d030 'x' <se r\377\377\377\377\377\377\377\377p\377\377\377\377\377\377\377\377te 200 fois>..., 
    num_elements=-2147483648) at /home/antoine/arrow/cpp/src/arrow/buffer.h:332
#4  0x00007fffc23d7f3e in arrow::BinaryBuilder::Append (this=0x7fffffffb4a0, 
    value=0x7fff3f01d030 'x' <se r\377\377\377\377\377\377\377\377p\377\377\377\377\377\377\377\377te 200 fois>..., length=-2147483648)
    at /home/antoine/arrow/cpp/src/arrow/builder.cc:1343
#5  0x00007fffc258d4b8 in arrow::BinaryBuilder::Append (this=0x7fffffffb4a0, 
    value=0x7fff3f01d030 'x' <se r\377\377\377\377\377\377\377\377p\377\377\377\377\377\377\377\377te 200 fois>..., length=-2147483648)
    at /home/antoine/arrow/cpp/src/arrow/builder.h:675
#6  0x00007fffc1f84923 in arrow::py::AppendObjectStrings (arr=0x7ffff7ecbee0, mask=0x0, offset=0, builder=0x7fffffffb4a0, 
    end_offset=0x7fffffffb480, have_bytes=0x7fffffffb430) at /home/antoine/arrow/cpp/src/arrow/python/numpy_to_arrow.cc:233
#7  0x00007fffc1f88d62 in arrow::py::NumPyConverter::ConvertObjectStrings (this=0x7fffffffbdd0)
    at /home/antoine/arrow/cpp/src/arrow/python/numpy_to_arrow.cc:859
#8  0x00007fffc1f8c22b in arrow::py::NumPyConverter::ConvertObjectsInfer (this=0x7fffffffbdd0)
    at /home/antoine/arrow/cpp/src/arrow/python/numpy_to_arrow.cc:1034
#9  0x00007fffc1f8d99c in arrow::py::NumPyConverter::ConvertObjects (this=0x7fffffffbdd0)
    at /home/antoine/arrow/cpp/src/arrow/python/numpy_to_arrow.cc:1135
#10 0x00007fffc1f85a2a in arrow::py::NumPyConverter::Convert (this=0x7fffffffbdd0)
    at /home/antoine/arrow/cpp/src/arrow/python/numpy_to_arrow.cc:504
#11 0x00007fffc1f9038d in arrow::py::NdarrayToArrow (pool=0x7fffc2a2d680 <arrow::default_memory_pool()::default_memory_pool_>, 
    ao=0x7ffff7ecbee0, mo=0x7ffff7d5ce90 <_Py_NoneStruct>, use_pandas_null_sentinels=true, type=..., out=0x7fffffffc0c0)
    at /home/antoine/arrow/cpp/src/arrow/python/numpy_to_arrow.cc:1577

[asfimport]

Antoine Pitrou / @pitrou:
By the way, is the enhancement requested in this ticket even doable with the current memory layout? If we create a chunked array and split the binary string in chunks, each string chunk will be visible as a separate logical array element, so the user won't see a 2GB string anymore, but for example two 1GB strings...

[asfimport]

Wes McKinney / @wesm:
OK, moving to 0.10.0.

We should raise an exception for a single string exceeding 2GB. We'll need to add a type for large binary / large strings to support this, see ARROW-750

[asfimport]

Left Screen:
Just wanted to mention, in case it was missed, but this example isn't a single large 2 GiB string. Each row in the data frame is a single byte. So it is a large array of small bytes.

Is 0.10.0 far away? Out of the box, using pyarrow for "big data" isn't really possible (assuming you have string data) until this is fixed. My hack fix was to make changes to pandas_compat.py:dataframe_to_arrays() so that it accepted a dictionary mapping column names to chunk sizes, and then I manually (and somewhat crudely) create a chunked_array. This is then passed to Table.from_arrays (mimicking what appears in table.pxi).

[asfimport]

Antoine Pitrou / @pitrou:

Just wanted to mention, in case it was missed, but this example isn't a single large 2 GiB string. Each row in the data frame is a single byte. So it is a large array of small bytes.

Oh, I see. I had misread the example (and my crash is on a different use case then). It's quite a weird way of storing binary strings, though? Your column is a column of Python objects, which under the hood appear to be numpy.int64 objects... So you're paying a huge overhead because of all those objects.

(to put in perspective, I have 16 GB RAM, but creating your dataframe swaps out...)

[asfimport]

Left Screen:
Yeah it's a contrived example, but think of a large data frame storing street addresses, or usernames, etc.

[asfimport]

Wes McKinney / @wesm:
I want to dig at this issue a bit before we cut the 0.9.0 release ... should have a little time later today

[asfimport]

Wes McKinney / @wesm:
This seems to be an off-by-one-error. In builder.cc, we are comparing with INT32_MAX - 1, in python/numpy_to_arrow.cc we are comparing with INT32_MAX. I made this a blocker for 0.9.0 as I think we can fix this by changing the bound in numpy_to_arrow.cc. Getting late here tonight so I will try to fix tomorrow morning before we cut an RC – the test case here is pretty swappy, we should be able to construct a better test case with some very large strings followed by some length-1 strings to hit the edge case at INT32_MAX

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 1740
#1740