[C++] Add support Parquet with Page Index

[asfimport]

A recent parquet-format release also added support for a Page Index making it possible to skip pages within a RowGroup.

This should be implemented by Apache Arrow.

Reporter: Malthe Borch
Assignee: Gang Wu / @wgtmac

Subtasks:

• [C++][Parquet] FileMetaData exposes page index metadata
• [C++][Parquet] Introduce ColumnIndex and OffsetIndex
• [C++][Parquet] Parquet page index read support
• Support predicate pushdown for page index in Parquet dataset scanner
• Enable writing page index by default

Related issues:

• [C++] Add index pages to the format to support efficient page skipping to parquet-cpp (is duplicated by)

_{Note: This issue was originally created as ARROW-10158. Please see the migration documentation for further details.}

[asfimport]

Uwe Korn / @xhochy:
[~malthe] Are you interesting in contributing this? Do you need pointers on what to do?

[asfimport]

Wes McKinney / @wesm:
I think this is a dup of PARQUET-1404

[asfimport]

Arun Balajiee Lekshmi Narayanan:
This is the placeholder pullrequest a2un#2

[asfimport]

shibei:
Hi @wgtmac , it is a good start to take advantage the power of parquet column index, is there any plan to adapt this in the read path?

[asfimport]

Gang Wu / @wgtmac:
Hi [~shibei], thanks for your interest!

This is still a work in progress. I am adding the read interface of page index via this PR: #14964

[asfimport]

shibei:
I don't mean reading column index data itself, but to support predicate pushdown with column index or row group level statistics, is there any plan?  @wgtmac  

[asfimport]

Gang Wu / @wgtmac:
[~shibei] Yes, that is part of my plan. But I cannot commit the time frame yet.

[mstebelev]

Hi @wgtmac, can you clarify, is there a way now to skip reading some pages using the data from PageIndex?
As far as I understood from the code, you can skip records and it can skip deserialization of the page, but the serialized data is read fully anyway. Is it right or I can avoid reading the page when skipping all records from them?

[wgtmac]

@mstebelev Quick answer is no. I plan to support this by adding an interface to pass requested row range to the reader so it gets the chance to leverage the page index to skip unnecessary pages. But I cannot commit the time frame as I am getting busy these days.

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.