source-ids is not supported on Iceberg v1,2#2114
Closed
rambleraptor wants to merge 5 commits intoapache:mainfrom
Closed
source-ids is not supported on Iceberg v1,2#2114rambleraptor wants to merge 5 commits intoapache:mainfrom
rambleraptor wants to merge 5 commits intoapache:mainfrom
Conversation
Fokko
requested changes
Jun 18, 2025
Contributor
Fokko
left a comment
There was a problem hiding this comment.
Hey @rambleraptor thanks for working on this, but I don't think this is the right approach. This breaks a lot of current APIs, and also requires users to pass in versions when creating PartitionFields and SortOrderFields.
A while ago, I've did some work on the spec to avoid having to carry through the table version to all the parsers, to exactly avoid this situation :) I think we should be good with just parsing either source-id or source-ids.
Contributor
Author
|
That makes a lot of sense for v3. But, it looks like |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1547
Rationale for this change
The field
source-idsis being introduced for Iceberg v3. According to the spec, it should not be supported by Iceberg v1 + v2. This introduces an error message to ensure that it cannot be used in those versions.Most of the code here is to plumb through the format_version to the field to do the actual version check.
Are these changes tested?
Unit tests included.
Are there any user-facing changes?
This does change the current support for
source-idson Iceberg v1/v2, but it shouldn't be supported in the first place.