Fix for 'Error reading Avro files containing array of struct with 2 fields #605'#618
Fix for 'Error reading Avro files containing array of struct with 2 fields #605'#618rdblue merged 2 commits intoapache:masterfrom
Conversation
|
|
||
| case ARRAY: | ||
| if (schema.getLogicalType() instanceof LogicalMap || AvroSchemaUtil.isKeyValueSchema(schema.getElementType())) { | ||
| if (schema.getLogicalType() instanceof LogicalMap && AvroSchemaUtil.isKeyValueSchema(schema.getElementType())) { |
There was a problem hiding this comment.
I think the solution should be a better check in isKeyValueSchema. If the logical type is LogicalMap, then it should not need additional requirements for the element type.
There was a problem hiding this comment.
If the logical type is LogicalMap, then it should not need additional requirements for the element type.
I think that makes sense. In that case. Maybe I should just simplify the if condition to
f (schema.getLogicalType() instanceof LogicalMap) and maybe make the isKeyValueSchema check as part of a PreconditionCheck?
There was a problem hiding this comment.
Yes, that sounds good.
This was originally an or condition because we had manifest files that were written with arrays of structs for maps, but without the map logical type. We still want to support reading files like that (at least in our version) but I think the right way to support it is to use an AvroSchemaWithType visitor introduced in #601 and checking whether the Iceberg type is a map.
|
+1 Thanks for fixing this, @rdsr! |
Releases 0.13.0.2-apple
2 participants
No description provided.