[Python] Expose more metadata in pyarrow.parquet.ParquetFile.metadata

[deanm0000]

Describe the enhancement requested

I'm not sure if this issue pertains to all implementations of arrow including pyarrow or just c++ but related to this #14870

I'm guessing it affects pyarrow as pq.ParquetFile.metadata.to_dict()['row_groups'][0]['columns'][0]['statistics'].keys()
doesn't have the min_value, max_value keys.

So the feature request is to include min_value and max_value in that metadata.

Additionally, I think there's metadata on whether or not a column is sorted (I might be confused on that point) but if there is it'd be good to see that too.

Component(s)

Python

[jorisvandenbossche]

@deanm0000 the statistics indeed don't have those exact min_value / max_value keys, but it does have min / max and min_raw / max_raw keys. Does those already do what you are looking for?
(it seems the _raw ones are only available as attribute on the object (eg meta.row_group(0).column(0).statistics.min_raw), and not in the dict)

[deanm0000]

So my thoughts are that some readers and optimizers use the "newer" min_value and max_value stats when they plan queries and filters. (Hopefully pyarrow.dataset is included in that or will be). I'd like a way to verify that my parquet files have those stats. Since the min and max stats are deprecated it seems fewer libraries are going to even look at those if they exist. As to the min_raw and max_raw, I've never heard of them so I'm not sure how valuable they are.

[jorisvandenbossche]

Starting from the next version (#34112), the min/max python attributes will check for min_value/max_value in the actual parquet metadata, if they are present, and otherwise fallback to the deprecated min/max values.

[deanm0000]

Does the writer write min_value/max_value and if so has it done that for a while?

[jorisvandenbossche]

Yes, AFAIK we have been writing those values for a long time. As a quick test using pyarrow 1.0 (from almost 3 years ago), writing a small table, and inspecting the thrift metadata of the file with parquet-tools, both min/max and min_value/max_value are set.