[Python] Regression memory issue when calling pandas.read_parquet

[asfimport]

I have a ~3MB parquet file with the next schema:

bag_stamp: timestamp[ns]
transforms_[]_.header.seq: list<item: int64>
  child 0, item: int64
transforms_[]_.header.stamp: list<item: timestamp[ns]>
  child 0, item: timestamp[ns]
transforms_[]_.header.frame_id: list<item: string>
  child 0, item: string
transforms_[]_.child_frame_id: list<item: string>
  child 0, item: string
transforms_[]_.transform.translation.x: list<item: double>
  child 0, item: double
transforms_[]_.transform.translation.y: list<item: double>
  child 0, item: double
transforms_[]_.transform.translation.z: list<item: double>
  child 0, item: double
transforms_[]_.transform.rotation.x: list<item: double>
  child 0, item: double
transforms_[]_.transform.rotation.y: list<item: double>
  child 0, item: double
transforms_[]_.transform.rotation.z: list<item: double>
  child 0, item: double
transforms_[]_.transform.rotation.w: list<item: double>
  child 0, item: double

 If I read it with pandas.read_parquet() using pyarrow 0.13.0 all seems fine and it takes no time to load. If I try the same with 0.14.0 or 0.14.1 it takes a lot of time and uses ~10GB of RAM. Many times if I don't have enough available memory it will just be killed OOM. Now, if I use the next code snippet instead it works perfectly with all the versions:

parquet_file = pq.ParquetFile(input_file)
tables = []
for row_group in range(parquet_file.num_row_groups):
    tables.append(parquet_file.read_row_group(row_group, columns=columns, use_pandas_metadata=True))
df = pa.concat_tables(tables).to_pandas()

Reporter: Francisco Sanchez

Related issues:

• [Python] too large memory cost using pyarrow.parquet.read_table with use_threads=True (is related to)

Original Issue Attachments:

• Memory_profile_0.13_rs.png
• Memory_profile_0.13.png
• Memory_profile_0.14.1_use_thread_false_rs.png
• Memory_profile_0.14.1_use_thread_FALSE.png
• Memory_profile_0.14.1_use_thread_true.png

_{Note: This issue was originally created as ARROW-6059. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
Can you provide a sample file that reproduces the issue?

[asfimport]

Francisco Sanchez:
For some reason I cannot upload a file here, it always fails. I have uploaded a dummy file here that reproduces the issue: http://s000.tinyupload.com/index.php?file_id=55835483534116457607

 

[asfimport]

Olivier Giboin:
I confirm a very similar issue:

• Context: Reading arrow table with pyarrow.parquet.read_table. When using 0.13 no memory error, ~6gb arrow RAM size when loaded, fast read.

• Context: Script ran in a VM with ~12gb free RAM, Windows, python3.7.x (conda)

• When reading the same parquet file with pyarrow 0.14.1 --> memory error (malloc failed, no more free ram availabe)

• Parquet file was generated by pyarrow.parquet.ParquetWriter

   

  I can provide full parquet file on secure channel if needed

[asfimport]

Francisco Sanchez:
I can confirm that in my case I also use pyarrow.parquet.ParquetWriter to build the parquet file, just in case it is relevant to the issue.

[asfimport]

Olivier Giboin:
-Btw this issue seems to be linked with https://issues.apache.org/jira/browse/ARROW-6060-

After further testing, I confirmed error occurs regardless use_threads = False/True –> not directly related to ARROW-6060

[asfimport]

Wes McKinney / @wesm:
I will try to investigate when I can, but as this software is developed largely by volunteers it's hard to predict how long a fix will take

[asfimport]

Francisco Sanchez:
[~ggGibs] I am not sure it is related to the one you mentioned, initially I also thought that and I tried to pass this argument set to False from the pandas call. It took more time but end up using the same amount of memory. I would think it is more related to ARROW-5965 but I don't have any evidence.

[asfimport]

Olivier Giboin:
Just did a test with use_threads = False -~~> same error.

-~~> correct, different symptoms than ARROW-6060, but same root cause?

 

Interestingly, the way memory gets allocated during read_table looks different between 0.13 and 0.14.1

v0.13 - smooth memory allocation (took ~40s to load file). Load successful

v0.14.1 - weird spiky memory allocation (shown here with use_threads = False - gets spikier when switching to True)

Load failed: I get a malloc error at the end of the command

[asfimport]

Wes McKinney / @wesm:
I think this is the same underlying cause as ARROW-6060. I traced the issue to the patch for ARROW-3762. We can confirm on this issue once that is resolved

[asfimport]

Wes McKinney / @wesm:
This should be resolved with the fix for ARROW-6060. If you can verify from master that would be helpful. If you run into more issues please reopen an issue