You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Basically, the patch provides following implementation:
Define class RowGroupPageIndexReader to read page index from a parquet row group. It internally leverages implementation from Apache Impala link to merge I/O chunks of page index in the same row group.
Define class PageIndexReader to create RowGroupPageIndexReader for each row group.
ParquetFileReader internally creates and caches a single PageIndexReader object and exposes it to the end user.
Limitation:
Reading page index from encrypted parquet file is not yet supported. It takes some effort to understand the specs and will be done in a separate patch after the completion of write logic of page index.
@wgtmac took a pass through mostly looking at interfaces, let me know what you think of the suggestions. Sorry for the delay, catching up after the holidays.
wgtmac
changed the title
ARROW-18434: [C++][Parquet] Parquet page index read support
GH-33596: [C++][Parquet] Parquet page index read support
Jan 12, 2023
I have added a reader test to cover the new interface. Now it is complete and ready to review. Any feedback is appreciated. @pitrou@emkornfield@wjones127
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
wgtmac
changed the title
Basically, the patch provides following implementation:
class RowGroupPageIndexReaderto read page index from a parquet row group. It internally leverages implementation from Apache Impala link to merge I/O chunks of page index in the same row group.class PageIndexReaderto createRowGroupPageIndexReaderfor each row group.ParquetFileReaderinternally creates and caches a singlePageIndexReaderobject and exposes it to the end user.Limitation: