iceberg: fix schema column ordering#4373
Conversation
Preserve schema registry column order when creating Iceberg tables. Column order was lost because buildSchemaWithResolver iterated over a map[string]any, which Go randomises. The fix reads the ordered field list from schema.Common metadata when present, falling back to sorted keys for determinism when it isn't.
|
Commits Review LGTM |
…ering Build-time column ordering now parses the schema metadata once at the call site and threads the parsed common through both ordering and per-field type resolution. The merge keeps metadata-declared fields first, falling back to a case-folded record key lookup so canonical lowercase metadata still matches arbitrarily-cased record keys, then appends any record keys that aren't in the metadata in sorted order so they're never silently dropped from create-table. Top-level metadata that declares two children differing only in case under case-insensitive mode is now rejected at create time, matching the existing guard for nested struct metadata.
In case-insensitive mode the build path now uses the metadata's casing for any top-level column whose record key only matched by case-folding. The record key is still used to look up the value, but the iceberg column lands with the metadata's name — matching the rule already in effect for nested struct fields supplied via schema metadata. The schema_metadata field description now spells out that record presence drives schema shape: metadata-only fields are not added to the table, and metadata controls ordering, naming, and types only for fields the record carries. Adds a CHANGELOG entry covering the column-ordering work on this branch.
|
Commits Review LGTM |
|
Commits Review LGTM |
2 participants
Preserve schema registry column order when creating Iceberg tables. Column order was lost because
buildSchemaWithResolveriterated over amap[string]any, which Go randomises. The fix reads the ordered field list fromschema.Commonmetadata when present, falling back to sorted keys for determinism when it isn't.