Skip to content

feat: Add conversion from FileMetaData to ParquetMetadata#1074

Merged
liurenjie1024 merged 5 commits intoapache:mainfrom
jonathanc-n:change-parquet-metadata-version
Mar 14, 2025
Merged

feat: Add conversion from FileMetaData to ParquetMetadata#1074
liurenjie1024 merged 5 commits intoapache:mainfrom
jonathanc-n:change-parquet-metadata-version

Conversation

@jonathanc-n
Copy link
Copy Markdown
Contributor

Which issue does this PR close?

What changes are included in this PR?

Add conversion from filemetadat to parquet metadata using thrift decode_metadata

Are these changes tested?

liurenjie1024
liurenjie1024 previously approved these changes Mar 13, 2025
Copy link
Copy Markdown
Contributor

@liurenjie1024 liurenjie1024 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jonathanc-n , LGTM! Just a minor concern about thrift depdenency, let's wait for @Xuanwo to take a look.

Comment thread Cargo.toml
Copy link
Copy Markdown
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most changes looks good to me. It's better if we can remove some unwarp().

Comment thread Cargo.toml
Comment thread crates/iceberg/src/writer/file_writer/parquet_writer.rs Outdated
Comment thread crates/iceberg/src/writer/file_writer/parquet_writer.rs Outdated
@liurenjie1024 liurenjie1024 dismissed their stale review March 14, 2025 01:38

We should remove the unwrap here.

@jonathanc-n
Copy link
Copy Markdown
Contributor Author

@Xuanwo @liurenjie1024 Should be fine now. Thanks for the review!

Copy link
Copy Markdown
Contributor

@liurenjie1024 liurenjie1024 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @jonathanc-n !

@liurenjie1024 liurenjie1024 merged commit 3cc1a68 into apache:main Mar 14, 2025
@jonathanc-n jonathanc-n deleted the change-parquet-metadata-version branch March 28, 2025 22:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consolidate methods of converting parquet file to data file builder.

3 participants