BinaryArrow Byte Limitation

[chebee7i]

I'm having some trouble converting a DataFrame into a table:

import numpy as np
import pandas as pd
import pyarrow as pa

x = list('1' * 2**31)
y = pd.DataFrame({'x': x})
t = pa.Table.from_pandas(y)
# ArrowInvalid: BinaryArrow cannot contain more than 2147483646 bytes, have 2147483647

This is for pyarrow 0.8.0. Should it be creating chunked arrays automatically for me? Ultimately, my goal is to write to parquet.

[wesm]

Yes, it should be creating chunked arrays. Can you open a JIRA for this?

[chebee7i]

Will do. Is there a simple workaround for the time being?

[chebee7i]

ARROW-2227

[wesm]

Closing in favor of tracking on JIRA