Crash for categorical column when exceeds 2GB on write

[jlwaldeck]

Environment

• pyarrow version: 0.7.1 - 0.8.0
• Python version: 3.6
• Operating System: Windows 8-10

Details

• Describe what you were trying to get done.

  Save large parquet file containing categorical column

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

import numpy as np
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq


def test_bytes_exceed_2gb_write():
    """Based on test from #867. Shows OK behavior for `str` column."""
    val = 'x' * (1 << 20)
    df = pd.DataFrame({
        'strings': np.array([val] * 4000, dtype=object)
    })

    table = pa.Table.from_pandas(df)

    assert table[0].data.num_chunks == 2
    pq.write_table(table, 'foo.parquet')


def test_categorical_exceed_2gb_write():
    """Based on test from #867. Shows Crash for `categorical` column."""
    val = 'x' * (1 << 20)
    df = pd.DataFrame({
        'strings': np.array([val] * 4000, dtype=object)
    })
    df['strings'] = df['strings'].astype('category')

    table = pa.Table.from_pandas(df)

    assert table[0].data.num_chunks == 1  # Only 1 block is being generated
    pq.write_table(table, 'foo.parquet')  # Fails here


test_bytes_exceed_2gb_write()
test_categorical_exceed_2gb_write()

• If there was a crash, please include the traceback here.

Traceback (most recent call last):
  File "C:\Users\some_user\Desktop\bug_categorical.py", line 39, in <module>
    test_categorical_exceed_2gb_write()
  File "C:\Users\some_user\Desktop\bug_categorical.py", line 35, in test_categorical_exceed_2gb_write
    pq.write_table(table, 'foo.parquet')  # Fails here
  File "C:\Anaconda\lib\site-packages\pyarrow\parquet.py", line 944, in write_table
    writer.write_table(table, row_group_size=row_group_size)
  File "C:\Anaconda\lib\site-packages\pyarrow\parquet.py", line 297, in write_table
    self.writer.write_table(table, row_group_size=row_group_size)
  File "_parquet.pyx", line 930, in pyarrow._parquet.ParquetWriter.write_table
  File "error.pxi", line 77, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: BinaryArray cannot contain more than 2147483646 bytes, have 2147483648

Similar to https://issues.apache.org/jira/browse/ARROW-1167 and also #1673

[jlwaldeck]

cc: @vmuriart

[n0mn0m]

I'm running into the same issue with similar python versioning and OS. I'll try to test this on Linux later. Happy to dig in on this if there is any direction here or on jira.

[vmuriart]

@alexhagerman I think you may find clues about this in the JIRA @jlwaldeck noted and also the related PR #867.

@jlwaldeck and I tested this across multiple versions and noticed that it fails when the categorical column is about to be written about. My uneducated guess is that changes from #867 are being by-passed since the column is originally compressed as a categorical, but right before doing the write, the table is expanding the column's array and once again trips over the same memory limitation (and crash).

We weren't able to replicate the issue without the write_table call

[n0mn0m]

@vmuriart I'm able to generate this on non categorical columns without a call to write_table. Maybe I should open a separate issue? Not sure. I've read the comments on the Jira ticket and I will begin to look at the file commit history. Also joined the slack channel in case that's a better place to get details.

Environment

• Windows 7
• Python 3.6
• pyarrow 0.8.0

import pyarrow as pa
import pyarrow.parquet as pq

parquet_file = pq.ParquetFile('sample.parquet')
parquet_file.metadata

<pyarrow._parquet.FileMetaData object at 0x000000000459B368>
  created_by: parquet-cpp version 1.3.2-SNAPSHOT
  num_columns: 9
  num_rows: 3241880
  num_row_groups: 1
  format_version: 1.0
  serialized_size: 10671

parquet_file.schema

<pyarrow._parquet.ParquetSchema object at 0x000000000958AF48>
col_one: INT64
col_two: INT64
col_three: BYTE_ARRAY UTF8
col_four: BYTE_ARRAY UTF8
col_five: BYTE_ARRAY UTF8
col_six: INT64
col_seven: BYTE_ARRAY UTF8
col_eight: BYTE_ARRAY UTF8
__index_level_0__: INT64

parquet_file.num_row_groups

1

parquet_file.read_row_group(0)

---------------------------------------------------------------------------
ArrowIOError                              Traceback (most recent call last)
<ipython-input-5-db281ae27c00> in <module>()
----> 1 parquet_file.read_row_group(0)

~\AppData\Local\conda\conda\envs\jupyter_notes\lib\site-packages\pyarrow\parquet.py in read_row_group(self, i, columns, nthreads, use_pandas_metadata)
    113             columns, use_pandas_metadata=use_pandas_metadata)
    114         return self.reader.read_row_group(i, column_indices=column_indices,
--> 115                                           nthreads=nthreads)
    116 
    117     def read(self, columns=None, nthreads=1, use_pandas_metadata=False):

_parquet.pyx in pyarrow._parquet.ParquetReader.read_row_group()

error.pxi in pyarrow.lib.check_status()

ArrowIOError: Arrow error: Invalid: BinaryArray cannot contain more than 2147483646 bytes, have 2147484540

[dmadeka]

@alexhagerman Any updates on this? I keep getting this error for big files too

[n0mn0m]

@dmadeka I don't have a fix update, but the Jira for this was moved from Arrow to Parquet you can see more information here: https://issues.apache.org/jira/browse/PARQUET-1239

[wesm]

Closing this issue. Let's continue discussion on JIRA