File Format API

[pvary]

Proposed Change

Iceberg currently supports 3 different file formats: Avro, Parquet, ORC. With the introduction of Iceberg V3 specification many new features are added to Iceberg. Some of these features like new column types, default values require changes at the file format level. The changes are added by individual developers with different focus on the different file formats. As a result not all of the features are available for every supported file format.
Also there are emerging file formats like Vortex [1] or Lance [2] which either by specialization, or by applying newer research results could provide better alternatives for certain use-cases like random access for data, or storing ML models.

Proposal document

https://docs.google.com/document/d/1sF_d4tFxJsZWsZFCyCL9ZE7YuI7-P3VrzMLIrrTIxds

Specifications

• Table
• View
• REST
• Puffin
• Encryption
• Other

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[prrao87]

Hi, thanks for including Lance in the list of proposed formats to support. I'd like to point out that the Lance page has now moved to lance.org, so it would be great to update the citation in the Google doc accordingly.

[pvary]

Hi, thanks for including Lance in the list of proposed formats to support. I'd like to point out that the Lance page has now moved to lance.org, so it would be great to update the citation in the Google doc accordingly.

Done

[PhilippLeser]

+1 for vortex :)

[d33bs]

+1 for Lance and Vortex support. Please implement this, it'd be amazing!

[pvary]

The draft Vortex PR is available here: #15915