[Python] pandas.read_parquet fails for wide parquet files and pyarrow 1.0.X

[asfimport]

I recently tried to update my pyarrow from 0.17.1 to 1.0.0 and I'm encountering a serious bug where wide DataFrames fail during pandas.read_parquet.  Small parquet files (m=10000) read correctly, medium files (m=40000) fail with a "Bus Error: 10", and large files (m=100000) completely hang.  I've tried python 3.8.5, pandas 1.0.5, pyarrow 1.0.0, and OSX 10.14.   

The driver code and output is below:

import pandas as pd
import numpy as np
import sys

filename = "test.parquet"
n = 10
m = int(sys.argv[1])
print(m)
x = np.zeros((n, m))
x = pd.DataFrame(x, columns=[f"A_{i}" for i in range(m)])
x.to_parquet(filename)
y = pd.read_parquet(filename, engine='pyarrow')

time python test_pyarrow.py  10000
real 0m4.018s user 0m5.286s sys 0m0.514s
time python test_pyarrow.py  40000
40000
Bus error: 10

 

On a pyarrow 0.17.1 environment, the 40,000 case completes in 8 seconds.  

This was cross-posted on the pandas tracker as well: pandas-dev/pandas#35846

Reporter: Kyle Beauchamp
Assignee: Joris Van den Bossche / @jorisvandenbossche

PRs and other links:

• GitHub Pull Request #8037

_{Note: This issue was originally created as ARROW-9827. Please see the migration documentation for further details.}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
[~kyleabeauchamp] thanks for moving the issue here.

In pyarrow 1.0, we switched the default for pyarrow.parquet.read_table (what pandas uses under the hood) to use the new Dataset API (which eg enables filtering row groups). When reading the file with 40,000 columns, I get a similar result for 1.0 as you got for 0.17 if I disable that:

In [3]: import pyarrow.parquet as pq   

In [4]: %time table = pq.read_table("test.parquet", use_legacy_dataset=True)                                                                                                                                       
CPU times: user 11.4 s, sys: 296 ms, total: 11.7 s
Wall time: 7.98 s

But without specifying use_legacy_dataset=True (so with using the default of use_legacy_dataset=False), it "completed" after more than 4 minutes, but then directly segfaulted upon return, for me for this case.

So that's certainly a temporary workaround for you to be able to upgrade to 1.0 to use this keyword. You can specify it from the pandas interface as well (and it will get passed through): pd.read_parquet("test.parquet", use_legacy_dataset=True)

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
As I suspected, this is due to the reading of metadata, and especially the column statistics. In the Dataset API, we by default parse all metadata including statistics, even when there is no filter specified.

When disabling the parsing of statistics, the file with 40,000 columns reads in 2.5 seconds (will open a PR with that).

Further, with the released version of pyarrow, using ParquetFile is actually the fastest option right now:

In [7]: %time table = pq.ParquetFile("test.parquet").read()                                                                                                                                                        
CPU times: user 7.89 s, sys: 253 ms, total: 8.14 s
Wall time: 4.5 s

Additional sidenote: the Parquet file format is generally not well suited for such wide tables, because of the column-heavy metadata.
For example, reading a Parquet file with the same number of values, but in long format (40,000 rows x 10 columns) reads in less than 10ms instead of in multiple seconds)

[asfimport]

Ben Kietzman / @bkietz:
Issue resolved by pull request 8037
#8037