[Python] Reading a dictionary column from Parquet results in disproportionate memory usage

[asfimport]

I'm using pyarrow to read a 40MB parquet file.

When reading all of the columns besides the "body" columns, the process peaks at 170MB.

Reading only the "body" column results in over 6GB of memory used.

I made the file publicly accessible: s3://dhavivresearch/pyarrow/demofile.parquet

 

 

Reporter: Daniel Haviv
Assignee: Wes McKinney / @wesm

Related issues:

• [Python] too large memory cost using pyarrow.parquet.read_table with use_threads=True (duplicates)
• [Python] too large memory cost using pyarrow.parquet.read_table with use_threads=True (is caused by)
• [C++] Read Parquet dictionary encoded ColumnChunks directly into an Arrow DictionaryArray (relates to)

_{Note: This issue was originally created as ARROW-5993. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
This isn't actually a bug but it's a symptom of fully materializing a densely dictionary-encoded column.

I'm addressing this in my patch for ARROW-3772 and follow up patches which will allow this data to be read directly as dictionary-encoded without blowing up memory. I'll leave this issue open so we can validate the the patch once I get it submitted

[asfimport]

Wes McKinney / @wesm:
[~danielil] can you make this file publicly accessible again? I'm getting "Access Denied" at that URL. I think you are probably hitting ARROW-6060 which was fixed but I can confirm by checking the file

[asfimport]

Wes McKinney / @wesm:
I was able to get the file from https://dhavivresearch.s3.amazonaws.com/pyarrow/demofile.parquet

[asfimport]

Wes McKinney / @wesm:
I confirmed that this is ARROW-6060. On master peak memory use reading the entire file is 85MB. In 0.14.1 it's a little under 6GB. This code can be used to reproduce

import pandas as pd
from pandas.util.testing import rands

import pyarrow as pa
import pyarrow.parquet as pq

import gc
class memory_use:
    
    def __init__(self):
        self.start_use = pa.total_allocated_bytes()        
        self.pool = pa.default_memory_pool()
        self.start_peak_use = self.pool.max_memory()
        
    def __enter__(self):
        return
    
    def __exit__(self, type, value, traceback):
        gc.collect()
        print("Change in memory use: {}"
              .format(pa.total_allocated_bytes() - self.start_use))
        print("Change in peak use: {}"
              .format(self.pool.max_memory() - self.start_peak_use))

with memory_use():
    table = pq.read_table('/home/wesm/Downloads/demofile.parquet')        

[asfimport]

Daniel Haviv:
Thanks Wes, amazing work :)

Is there an estimated release date for 0.15?

 

[asfimport]

Neal Richardson / @nealrichardson:
0.15 is probably still a couple weeks away. You can follow the issue burndown at https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.15.0+Release