Migration of ORC Backed Timestamp Without Zone Tables yields Timestamp With Zone columns

[RussellSpitzer]

A user was attempting to convert an ORC backed external table in hive to a Iceberg table using the migrate command but was immediately met with a "Can not promote TIMESTAMP to TIMESTAMP" error. This occurs because our Spark -> Iceberg conversion code always converts to a Timestamp.withZone.

This is pretty easy to reproduce with

spark.sql("CREATE EXTERNAL TABLE mytable (foo timestamp) STORED AS orc LOCATION '/Users/russellspitzer/Temp/foo'")

spark.sql("INSERT INTO mytable VALUES (now())")

spark.sql("CALL spark_catalog.system.migrate('mytable')")


spark.sql("SELECT * FROM mytable")

java.lang.IllegalArgumentException: Can not promote TIMESTAMP type to TIMESTAMP
	at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:441)
	at org.apache.iceberg.orc.ORCSchemaUtil.buildOrcProjection(ORCSchemaUtil.java:301)
	at org.apache.iceberg.orc.ORCSchemaUtil.buildOrcProjection(ORCSchemaUtil.java:275)
	at org.apache.iceberg.orc.ORCSchemaUtil.buildOrcProjection(ORCSchemaUtil.java:258)

I attempted to to just modify the Iceberg table so that it correctly typed the columns as Timestamp.withoutZone() but this leads to #2244

[atifiu]

@RussellSpitzer So what is the solution to this issue . I am getting the exact same error when trying to rewrite data files which I have added to iceberg table form hive using add_files.

[atifiu]

@RussellSpitzer I have already tried adding below but no impact
.config("spark.sql.iceberg.handle-timestamp-without-timezone","true")

[pravin1406]

@atifiu @RussellSpitzer Hi We are facing the same issue, tried making some code changes to ORCschemautil to bypass the error but it gives the incorrect timestamp as output . How did you guys go about it ?

[RussellSpitzer]

Can you just convert timestamp to timestampz after the fact? The underlying long is identical, just the display representation should be different