Fix casting issues when joining two Iceberg tables together on Tez #1740
Conversation
| return new Date((Date) o); | ||
| } else { | ||
| return o; | ||
| } |
There was a problem hiding this comment.
Why would this be called to copy a non-Date object?
There was a problem hiding this comment.
On Tez, this is called with LocalDate. On MR we get Date, which is just a wrapper around LocalDate.
There was a problem hiding this comment.
Why not support both representations that are possibly passed in by Hive?
There was a problem hiding this comment.
Nevermind, I see that you did.
| joinTables("decimaltable", "decimal_col", Types.DecimalType.of(3, 1)); | ||
| joinTables("timestamptable", "timestamp_col", Types.TimestampType.withZone()); | ||
| joinTables("binarytable", "binary_col", Types.BinaryType.get()); | ||
| joinTables("datetable", "date_col", Types.DateType.get()); |
There was a problem hiding this comment.
Could you rewrite this as a new test case with a list of types to test?
We consider it a best practice to start new test cases rather than adding to existing, complete cases. Each test method is run independently so you see more of the failures that way. By making longer test methods with more than one case, failures can prevent other tests from even running.
I think it would be fine to use a loop over types in a single new case, since most of the code is the same for these.
There was a problem hiding this comment.
Thanks @rdblue for the review. I created a new test case.
lcspinter
force-pushed
the
CDPD-18710
branch
2 times, most recently
from
f41d5fa to
217b0b2
Compare
lcspinter
force-pushed
the
CDPD-18710
branch
from
217b0b2 to
f667f64
Compare
2 participants
The following casting exceptions were observed when joining two Iceberg tables on Tez:
java.math.BigDecimal cannot be cast to org.apache.hadoop.hive.common.type.HiveDecimal
java.time.OffsetDateTime cannot be cast to org.apache.hadoop.hive.common.type.Timestamp
java.nio.HeapByteBuffer cannot be cast to [B
java.time.LocalDate cannot be cast to org.apache.hadoop.hive.common.type.Date
Doesn't happen with Iceberg table - non-Iceberg table joins, or with order by queries on a single table.