[Python] Pandas to_feather no longer works - runs out of memory

[asfimport]

Since upgrading to 4.0.1 writing to feather files with the pandas to_feather method uses up far, far more memory.

For reference I have a dataframe that is around 10gb in size, 25 million rows. Writing a feather file took around 3-4gb of memory in pyarrow versions up to 3.0.0. As of 4.0.1 I don't know how much memory it will take to successfully write - I tried running on a 120gb AWS machine, and that wasn't sufficient.

I can't provide the dataframe, but I can give an outline of the types / sizes of the columns:

size (bytes),type
206663144,int64
206663144,int64
206663144,float64
206663144,float64
2882448709,object
5813798687,object
206663144,float64
206663144,int64
206663144,int64
206663144,int64
206663144,int64
206663144,float64

Environment: Linux
Reporter: Roland Swingler

Related issues:

• [C++][Python] Converter::Extend gets stuck in infinite loop causing OOM if values don't fit in single chunk (duplicates)

_{Note: This issue was originally created as ARROW-13014. Please see the migration documentation for further details.}

[asfimport]

yibocai#1:
Duplicates ARROW-12983 ?
cc @lidavidm

 

[asfimport]

David Li / @lidavidm:
Hmm, I'll have to give this a try in the morning, but it may be a duplicate, yes - I'd guess this goes through the same code path.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
[~knaveofdiamonds] you have two "object" dtype columns. Do those contain strings? (in which it indeed might be related to ARROW-12983)

[asfimport]

Roland Swingler:
@jorisvandenbossche - yes those columns contain strings (of reasonable length as well in some cases - i.e. one of the columns is a json-serialized array of numbers, so could be relatively long) .

[asfimport]

David Li / @lidavidm:
In that case yes, I think it's a duplicate. Observe:

>>> import pyarrow as pa
>>> import pandas as pd
>>> pa.__version__
'4.0.1'
>>> pd.__version__
'1.2.4'
>>> pd.DataFrame({"str": ["a" * 16384] * (2**17 - 1)}).reset_index().to_feather('/tmp/foo.feather')
# Finishes successfully
>>> pd.DataFrame({"str": ["a" * 16384] * (2**17 + 1)}).reset_index().to_feather('/tmp/foo.feather')
# Hangs while memory usage shoots up

[asfimport]

David Li / @lidavidm:
The key thing is that due to the linked bug, a string column can't have either more than 2 GiB of data or 2**31 rows when converted from Python/Pandas/etc to PyArrow - so if you could somehow chunk it (either into multiple files, or by slicing and manually writing the file) that should let you get around the issue.

[asfimport]

Roland Swingler:
@lidavidm and others - great - thanks for the explanation & fast responses.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Another possible workaround for now (instead of chunking) is to manually specify the schema when converting from pandas to arrow, and then specify "large_string" for the string columns