Spark: Pass correct types to get data from InternalRow #999
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Thanks @rdblue . I'll look into it tomorrow! |
Contributor
Author
|
Thanks, @rdsr! Good for you to review since we will need to do the same thing for the ORC writers. |
rdsr
reviewed
May 6, 2020
spark/src/main/java/org/apache/iceberg/spark/data/AvroWithSparkSchemaVisitor.java
Show resolved
Hide resolved
spark/src/main/java/org/apache/iceberg/spark/data/AvroWithSparkSchemaVisitor.java
Show resolved
Hide resolved
spark/src/main/java/org/apache/iceberg/spark/data/ParquetWithSparkSchemaVisitor.java
Outdated
Show resolved
Hide resolved
spark/src/main/java/org/apache/iceberg/spark/data/ParquetWithSparkSchemaVisitor.java
Show resolved
Hide resolved
Contributor
|
LGTM. Minor comments |
Contributor
Author
|
Thanks, @rdsr! I've fixed the things you pointed out. |
rdsr
approved these changes
May 6, 2020
Contributor
rdsr
left a comment
rdsr
left a comment
There was a problem hiding this comment.
+1. Once the build goes through
rodmeneses
pushed a commit
to rodmeneses/iceberg
that referenced
this pull request
Feb 19, 2024
szehon-ho
pushed a commit
to szehon-ho/iceberg
that referenced
this pull request
Sep 16, 2024
rodmeneses
pushed a commit
to rodmeneses/iceberg
that referenced
this pull request
Jun 23, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixes a problem with Spark 3.0 CTAS queries that use tinyint or smallint types. When Iceberg converts a Dataset schema, it promotes both smaller integers to
int. Normally, Spark will insert casts in the analyzer so that the values are ints, but during a CTAS query, the table is created and the values may be passed as short or byte in the rows passed to Iceberg.The problem happens when Iceberg accesses values from
InternalRow. Before this commit, Iceberg would use the table's type to fetch a value, causing unsafe rows to return a corrupted byte or short value because 4 bytes had been read instead of 1 or 2.The fix is to keep track of the Dataset schema and use it when accessing fields. This required building visitors for Avro and Parquet that traverse a Spark schema with a file schema.