[Python] ListArray has trouble with sizes greater than kMaximumCapacity

[asfimport]

When creating a Pandas dataframe with lists as elements as a column the following error occurs when converting to a pyarrow.Table object.

Traceback (most recent call last):
File "arrow-2227.py", line 16, in <module>
arr = pa.array(df['strings'], from_pandas=True)
File "array.pxi", line 177, in pyarrow.lib.array
File "error.pxi", line 77, in pyarrow.lib.check_status
File "error.pxi", line 77, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: BinaryArray cannot contain more than 2147483646 bytes, have 2147483647

The following code was used to generate the error (adapted from ARROW-2227):

import pandas as pd
import pyarrow as pa

# Commented lines were used to test non-binary data types, both cause the same error
v1 = b'x' * 100000000
v2 = b'x' * 147483646
# v1 = 'x' * 100000000
# v2 = 'x' * 147483646

df = pd.DataFrame({
     'strings': [[v1]] * 20 + [[v2]] + [[b'x']]
     # 'strings': [[v1]] * 20 + [[v2]] + [['x']]
})
arr = pa.array(df['strings'], from_pandas=True)
assert isinstance(arr, pa.ChunkedArray), type(arr)

Code was run using Python 3.6 with PyArrow installed from conda-forge on macOS High Sierra.

Reporter: Bryant Menn
Assignee: Krisztian Szucs / @kszucs

Related issues:

• [C++][Python] Refactor python to arrow conversions based on a reusable conversion API (is fixed by)
• [Python] Produce chunked arrays for nested types in pyarrow.array (relates to)

_{Note: This issue was originally created as ARROW-2367. Please see the migration documentation for further details.}

[asfimport]

Krisztian Szucs / @kszucs:
What's the expected output? It should be able to hold 2^31 - 1 bytes?

According to my investigation the limit is set to ((2^31) - 1) - 1.

arrow/cpp/src/arrow/builder.h

Line 44 in f9c0701

constexpr int64_t kBinaryMemoryLimit = std::numeric_limits<int32_t>::max() - 1;

[asfimport]

Uwe Korn / @xhochy:
The expected output would be that the resulting pyarrow.Table has a ChunkedArray instance for the respective column that consists of multiple chunks. Currently we only generate a single pa.Array and wrap that simply in a pa.ChunkedArray. Thus we are limited to what a single pa.Array can hold.

This should only happen on pa.Table.from_pandas() though. When a user calls pa.array() the current error is expected behaviour. We could add a second method pa.chunked_array() that would return a ChunkedArray with more than one chunk if necessary.

[asfimport]

Krisztian Szucs / @kszucs:
Got it now Uwe, thanks! Just saw the assertion line, totally missed it.

[asfimport]

Bryant Menn:
So would the ideal implementation then create pa.chunked_array() and have pa.Table.from_pandas() call that instead or would there be some criteria to decide between pa.array() (I believe this is the current flow for pa.Tablefrom_pandas()) and pa.chunked_array()? I was trying to follow the process implemented for BinaryArray conversion to ChunkedArray but not finished working through and understanding it..

[asfimport]

Uwe Korn / @xhochy:
pa.chunked_array() would probably call the same C++ code as pa.Table.from_pandas() does. There are some slight difference between the creation of a Table from a DataFrame and a single Array from a Series/NumPy Array.

An important point is that a Table is a collection of Columns that are made up of a ChunkedArray each where each CunkedArray consists of one or more Arrays. So the change in pa.Table.from_pandas() will not affect user-facing code. pa.chunked_array() would be introducing a whole new API as we don't want pa.array() to return different classes depending on the size of the input.

[asfimport]

Wes McKinney / @wesm:
There were some improvements around chunked data handling. I'll take a peek at this to see what needs to be done

[asfimport]

Wes McKinney / @wesm:
This appears to be core-dumping right now

In [1]: paste                                                                                     
import pandas as pd
import pyarrow as pa

# Commented lines were used to test non-binary data types, both cause the same error
v1 = b'x' * 100000000
v2 = b'x' * 147483646
# v1 = 'x' * 100000000
# v2 = 'x' * 147483646

df = pd.DataFrame({
     'strings': [[v1]] * 20 + [[v2]] + [[b'x']]
     # 'strings': [[v1]] * 20 + [[v2]] + [['x']]
})

## -- End pasted text --

In [2]: arr = pa.array(df['strings'], from_pandas=True)                                           

In [3]: arr                                                                                       
Out[3]: WARNING: Logging before InitGoogleLogging() is written to STDERR
F0211 20:23:23.905639 31682 array.cc:128]  Check failed: (offset) <= (data.length) 
*** Check failure stack trace: ***
Aborted (core dumped)

I'll dig in a bit

[asfimport]

Wes McKinney / @wesm:
The case where the child builder yields chunked children is not handled, as already noted in the code:

https://github.com/apache/arrow/blob/master/cpp/src/arrow/python/python_to_arrow.cc#L611

I'll try to work out a patch

[asfimport]

Wes McKinney / @wesm:
Oof, ok, this is complicated.

We need to return a chunked array in the example above, where each chunk is a ListArray. The serious problem here is that a binary overflow can occur in the middle of a list element.

So we need to determine if appending the list values (e.g. an array slot's list of strings) is going to overflow the builder, and in that case start the next array chunk.

This is made more complicated by the SeqConverter initialization pattern in python_to_arrow.cc.

I'll leave this issue in 0.13 but it feels like a solid 2-3 hours of refactoring to get things sorted out

[asfimport]

Wes McKinney / @wesm:
This is nearly the same issue as ARROW-6281. I'll leave both open for now