[C++] Parquet arrow::Table reads error when overflowing capacity of BinaryArray

[asfimport]

1. When reading a parquet file with binary data > 2 GiB, we get an ArrowIOError due to it not creating chunked arrays. Reading each row group individually and then concatenating the tables works, however.

    
   {code:java}
   import pandas as pd
   import pyarrow as pa
   import pyarrow.parquet as pq

   x = pa.array(list('1' * 2**30))

   demo = 'demo.parquet'

   def scenario():
   t = pa.Table.from_arrays([x], ['x'])
   writer = pq.ParquetWriter(demo, t.schema)
   for i in range(2):
   writer.write_table(t)
   writer.close()

   pf = pq.ParquetFile(demo)
   

2. pyarrow.lib.ArrowIOError: Arrow error: Invalid: BinaryArray cannot contain more than 2147483646 bytes, have 2147483647
   t2 = pf.read()

3. Works, but note, there are 32 row groups, not 2 as suggested by:

4. https://arrow.apache.org/docs/python/parquet.html#finer-grained-reading-and-writing
   tables = [pf.read_row_group(i) for i in range(pf.num_row_groups)]
   t3 = pa.concat_tables(tables)

   scenario()
   {code}

Reporter: Left Screen
Assignee: Ben Kietzman / @bkietz

Related issues:

• [Python] Error with errno 22 when loading 3.6 GB Parquet file (is duplicated by)
• [Python] ArrowIOError: Arrow error: Capacity error during read (is duplicated by)
• [Python] Table.from_pandas does not create chunked_arrays. (relates to)
• [Python] read_row_group fails with Nested data conversions not implemented for chunked array outputs (is related to)
• [C++] Add chunked builder classes (is related to)

PRs and other links:

• GitHub Pull Request #3171
• GitHub Pull Request #4695
• GitHub Pull Request #5312
• Apache Arrow Issue 1677

_{Note: This issue was originally created as ARROW-3762. Please see the migration documentation for further details.}

[asfimport]

Left Screen:
Related ticket is not code-related, but workflow-related in terms of reading/writing binary data

[asfimport]

Alex Hagerman / @n0mn0m:
I think these may be related? #1677

[asfimport]

Wes McKinney / @wesm:
I moved this issue to Apache Parquet because that's where the fix will need to be implemented. Thank you!

[asfimport]

Jakub Okoński:
Is there any update on this? How hard would the fix be? Is it something a newcomer can contribute?

[asfimport]

Wes McKinney / @wesm:
It's hard to say but things are going to be much easier once the codebase merge #2453 is done

[asfimport]

Jason Kiley:
Like someone mentioned on the Github issue, I also see this consistently and without any categorical columns in my (pandas) dataframes. In my case, I tend to have one large text column (the body text of documents, where a row is an individual document with various columns for its components and associated metadata).

[asfimport]

Wes McKinney / @wesm:
This will get fixed eventually but it's hard to predict when.

[asfimport]

Wes McKinney / @wesm:
I just moved this issue back to Apache Arrow since it's more Arrow-related and the codebases are now one

[asfimport]

Jason Kiley:
No worries; I know there's a ton going on. When I ran into it again yesterday, I looked up the issues and noticed that there were some differences in what folks were reporting (e.g. categoricals), so I thought it might help to point out that I see it without them (and with a little context about when I do and what the data looks like). Sorry if I'm the motivation for your tweet, but my intent was to add information.

[asfimport]

Wes McKinney / @wesm:
No – there have been lots of cases where people have written on bug reports (not necessarily in Apache Arrow) things "This is really causing a problem for me, when will it be fixed?" which you did not do, so thank you =) One time the comment was "This is a deal breaker for me"

[asfimport]

Wes McKinney / @wesm:
I started working on this. Since we have some other places where we deal with binary / string builder overflows, I'm going to make a "chunked binary" builder class that composes the base builders. This can also be used to resolve ARROW-2970

[asfimport]

Wes McKinney / @wesm:
This change is going to be pretty ugly because most of the APIs in parquet::arrow::FileReader assume that the result of a NextBatch call is a contiguous, non-chunked array

https://github.com/apache/arrow/blob/master/cpp/src/parquet/arrow/reader.cc#L218

To fix this we have to change pretty much every place where we return an Array to instead return a ChunkedArray. This is a lot of code to refactor.

This issue is affecting a lot of people so I'm going to try to grind through it tomorrow and see how far I can get.

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 3171
#3171

[asfimport]

Marcin:
I still see this error when using 0.13.0, also tested with 0.12.0.

 

Is this a regression, or has never been fixed?

[asfimport]

Wes McKinney / @wesm:
I reopened the issue at tagged with 0.14.0 so we can look into what's going on