Skip to content

Conversation

@rzhang10
Copy link
Contributor

@rzhang10 rzhang10 commented Apr 12, 2022

This PR is the sequel to #4525, it implements the default value reading semantics specified by spec #4301, to read Avro and ORC format backed files/tables in Spark.

It made the core change in BaseDataReader to support converting default values to Spark-expected constant value objects.

For ORC, it extends the OrcSchemaWithTypeVisitor to have a default-value-aware visitor to build the tree of readers based on the idToConstant map, cooperating with changes in ORCSchemaUtil.buildOrcProjection to make things work. The logic for ORC vectorized reader was also implemented.

In Avro, similar implementation techniques were used, extending and modifying AvroSchemaWithTypeVisitor and AvroSchemaUtil.buildAvroProjection respectively.

@rdblue Please take a look, Thanks!

@github-actions github-actions bot added the core label Apr 14, 2022
@github-actions
Copy link

github-actions bot commented Aug 8, 2024

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions bot added the stale label Aug 8, 2024
@github-actions
Copy link

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions bot closed this Aug 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant