ArrowIOError: Arrow error: Capacity error

[MarkiesFredje]

Hi,

I'm using pyarrow 0.10

I have a dataframe about 90GB size in memory, with one object column contain strings up to 27 characters max.

basket_plateau.to_parquet("basket_plateau.parquet", compression=None) writes this file to disk just fine
basket_plateau = pd.read_parquet("basket_plateau.parquet") fails however.

ArrowIOError: Arrow error: Capacity error: BinaryArray cannot contain more than 2147483646 bytes, have 2147483655

I can reproduce this exact same error when I use pyarrow directly:
pq.write_table(pa.Table.from_pandas(basket_plateau), "basket_plateau.parquet")
basket_plateau= pq.read_table("basket_plateau.parquet")

Kr.
Fred

[wesm]

Hi, can you open a JIRA? If you can write down a minimal reproduction that would also be helpful. Thanks!

[MarkiesFredje]

Created ARROW-3139

[eromoe]

Still got ArrowIOError: Arrow error: Capacity error: BinaryArray cannot contain more than 2147483646 bytes, have 2147483664 .

pyarrow==0.11.1

[wesm]

The bug isn't fixed yet. See JIRA

[gwerbin]

I just got this same error. Pyarrow 0.11.1 and Pandas 0.23.4. Would it at least be possible to indicate in the error message which column is the problematic one?

[wesm]

Let's continue discussion on JIRA