[C++][Parquet] Tracking issue for cross-implementation LZ4 Parquet compression compatibility

[asfimport]

per PARQUET-1878, it seems that there are still problems with our use of LZ4 compression in the Parquet format. While we should fix this (the Parquet specification and our implementation of it), we may need to disable use of LZ4 compression until the appropriate compatibility testing can bed one.

Reporter: Wes McKinney / @wesm
Assignee: Antoine Pitrou / @pitrou

Subtasks:

• [C++][Parquet] Disable writing files with LZ4 codec

Related issues:

• [C++] lz4 codec is not compatible with Hadoop Lz4Codec (fixes)
• [C++] Fix reading LZ4-compressed Parquet files produced by Java Parquet implementation (relates to)
• Support LZ4_RAW codec (is related to)

PRs and other links:

• GitHub Pull Request #9244
• GitHub Pull Request #16

_{Note: This issue was originally created as ARROW-9177. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
Do you think there's something else remaining to do here? @wesm

[asfimport]

Wes McKinney / @wesm:
Seems like it is resolved now

[asfimport]

Steve M. Kim:
I think that this issue is still not resolved. Using pyarrow 2.0.0, I can not read a parquet file that was written with the lz compression using Java parquet-mr 1.10.1.

Metadata for the Parquet file:

$ parquet-tools meta /tmp/b3caf545-3a3e-4c3e-9542-9ecbca341306/e86fa357-6e1f-4acd-945c-59d558a9434a
file:        file:/tmp/b3caf545-3a3e-4c3e-9542-9ecbca341306/e86fa357-6e1f-4acd-945c-59d558a9434a
creator:     parquet-mr version 1.10.1 (build a89df8f9932b6ef6633d06069e50c9b7970bebd1)

file schema:
--------------------------------------------------------------------------------
c1:          REQUIRED BINARY R:0 D:0
c0:          REQUIRED INT64 R:0 D:0
c2:          REQUIRED INT32 R:0 D:0
c3:          REQUIRED BINARY R:0 D:0
v1:          OPTIONAL INT64 R:0 D:1

row group 1: RC:3330100 TS:33837867 OFFSET:4
--------------------------------------------------------------------------------
c1:           BINARY LZ4 DO:0 FPO:4 SZ:27496/26246/0.95 VC:3330100 ENC:PLAIN_DICTIONARY,BIT_PACKED ST:[min: 0x4151556B4E6B454A7156675642466570637A556C5067704746686D5354424276766866726F73616370, max: 0x666F58594E7856684F4765684265714C58785A5464446B76526C4D5358635968576A586E494B727059, num_nulls: 0]
c0:           INT64 LZ4 DO:0 FPO:27500 SZ:4434451/5905528/1.33 VC:3330100 ENC:PLAIN_DICTIONARY,BIT_PACKED ST:[min: 0, max: 8639, num_nulls: 0]
c2:           INT32 LZ4 DO:0 FPO:4461951 SZ:4401/839852/190.83 VC:3330100 ENC:PLAIN_DICTIONARY,BIT_PACKED ST:[min: 0, max: 2, num_nulls: 0]
c3:           BINARY LZ4 DO:0 FPO:4466352 SZ:2645/423441/160.09 VC:3330100 ENC:PLAIN_DICTIONARY,BIT_PACKED ST:[min: 0x41, max: 0x42, num_nulls: 0]
v1:           INT64 LZ4 DO:0 FPO:4468997 SZ:26748724/26642800/1.00 VC:3330100 ENC:RLE,PLAIN,BIT_PACKED ST:[min: -9223371059640220560, max: 9223371048678785692, num_nulls: 0]

row group 2: RC:1698380 TS:17291569 OFFSET:31217721
--------------------------------------------------------------------------------
c1:           BINARY LZ4 DO:0 FPO:31217721 SZ:14103/13453/0.95 VC:1698380 ENC:PLAIN_DICTIONARY,BIT_PACKED ST:[min: 0x666F58594E7856684F4765684265714C58785A5464446B76526C4D5358635968576A586E494B727059, max: 0x7847426D495659786A7277477A756C4A4172586C66724551435547474145716A716A675342426C6761, num_nulls: 0]
c0:           INT64 LZ4 DO:0 FPO:31231824 SZ:2272242/3045717/1.34 VC:1698380 ENC:PLAIN_DICTIONARY,BIT_PACKED ST:[min: 0, max: 8639, num_nulls: 0]
c2:           INT32 LZ4 DO:0 FPO:33504066 SZ:2286/428364/187.39 VC:1698380 ENC:PLAIN_DICTIONARY,BIT_PACKED ST:[min: 0, max: 2, num_nulls: 0]
c3:           BINARY LZ4 DO:0 FPO:33506352 SZ:1413/215996/152.86 VC:1698380 ENC:PLAIN_DICTIONARY,BIT_PACKED ST:[min: 0x41, max: 0x42, num_nulls: 0]
v1:           INT64 LZ4 DO:0 FPO:33507765 SZ:13642061/13588039/1.00 VC:1698380 ENC:RLE,PLAIN,BIT_PACKED ST:[min: -9223351962781870274, max: 9223365078578074233, num_nulls: 0] 

What happen when I try to read this file with pyarrow.parquet:

Python 3.6.11 | packaged by conda-forge | (default, Jul 28 2020, 23:15:00)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow as pa
>>> pa.__version__
'2.0.0'
>>> import pyarrow.parquet as pq
>>> pq.read_table('/tmp/b3caf545-3a3e-4c3e-9542-9ecbca341306/e86fa357-6e1f-4acd-945c-59d558a9434a')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/skim/.local/Miniconda3-py38_4.8.3-Linux-x86_64/envs/pyarrow/lib/python3.6/site-packages/pyarrow/parquet.py", line 1639, in read_table
    use_pandas_metadata=use_pandas_metadata)
  File "/home/skim/.local/Miniconda3-py38_4.8.3-Linux-x86_64/envs/pyarrow/lib/python3.6/site-packages/pyarrow/parquet.py", line 1517, in read
    use_threads=use_threads
  File "pyarrow/_dataset.pyx", line 405, in pyarrow._dataset.Dataset.to_table
  File "pyarrow/_dataset.pyx", line 2262, in pyarrow._dataset.Scanner.to_table
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
OSError: IOError: Corrupt Lz4 compressed data. 

[asfimport]

Steve M. Kim:
#7789 looks to me like a reasonable change, but I haven't read it very carefully to discern whether there was a subtle bug that would explain the incompatibility that I observe.

[asfimport]

Antoine Pitrou / @pitrou:
[~chairmank] Would you mind posting the file somewhere?

[asfimport]

Steve M. Kim:
I wrote a small program that generates a simplified Parquet file to show this incompatibility.

https://github.com/chairmank/arrow-9177-example

An example file can be downloaded directly from

https://github.com/chairmank/arrow-9177-example/blob/7685f21ea4acad2d8a834e6890ea9293406da3ee/561120a3094ee4513ba619b518c7a6093fe4e38398219ad172fb75373c3360b8.parquet

This example file has 10000 rows. By bisection search, I found that the "Corrupt LZ4 compressed data" error occurred when there were 6528 rows, but not when there were fewer rows. This is a clue that the problem is somehow related to data size.

 

[asfimport]

Antoine Pitrou / @pitrou:
Thank you [~chairmank], this helps a lot! Indeed it seems we misunderstood the undocumented Hadoop-LZ4-framing format :-(

[asfimport]

Antoine Pitrou / @pitrou:
[~chairmank] Can you post the decoded values in the file somewhere? At least the N first and last.

[asfimport]

Steve M. Kim:
These are the decoded values as line-delimited JSON:

 

https://github.com/chairmank/arrow-9177-example/blob/3a169e32701939de64a8ecafb155cb0b730cd8d8/561120a3094ee4513ba619b518c7a6093fe4e38398219ad172fb75373c3360b8_decoded.jsonl

[asfimport]

Antoine Pitrou / @pitrou:
Great, thank you. I can confirm that the PR for ARROW-11301 reads the file properly.

Depending on specifics of the release procedure, it may go into 3.0.0 or 4.0.0 (or perhaps a hypothetical 3.0.1).