Missing handling of Timestamp Without Timezone type

[RussellSpitzer]

Spark currently possess no readers for the Timestamp.withoutZone() type but is able to create tables with this schema. These tables if you attempt to read them from Spark will error out.

Currently in master the only reader for this type that I can see is for the generic case

iceberg/data/src/main/java/org/apache/iceberg/data/orc/GenericOrcReader.java

Lines 116 to 117 in 631efec

	case TIMESTAMP:
	return GenericOrcReaders.timestamps();

Other systems if they hit a table with this type will fail immediately since they do not have valid readers.

This is a bit troubling because this column type is used by default when non-iceberg ORC writers make new files, for example:

spark.sql("CREATE EXTERNAL TABLE mytable (foo timestamp) location '/Users/russellspitzer/Temp/foo'")

spark.sql("INSERT INTO mytable VALUES (now())")


Creates files

File Version: 0.12 with ORC_135
Rows: 1
Compression: SNAPPY
Compression size: 262144
Calendar: Julian/Gregorian
Type: struct<foo:timestamp>

The non-iceberg Spark and Hive Orc readers and writers have no problem dealing with these files but, if an iceberg table is created and these files are added to it then they are unreadable by Iceberg's orc readers and writers.

There is also a related problem with Migrate -- @RussellSpitzer Add Link Here

[RussellSpitzer]

We decided in the last Sync that a reasonable solution here would be for Spark to read both timezone types as timestamp with timezone with a flag disabling this behavior by default. If the flag is disabled an error message would alert the user that Spark cannot handle the two different types and will always display the local timezone, but the underlying data would be correct.

[javsanbel2]

Hello @RussellSpitzer! I am having a similar issue and I agree to your solution as well. Did you continue working on this? I would like to know more about your approach. How would this work in practice?

[RussellSpitzer]

There is an active pr to fix this, please help with the review if you are interested. Basically everywhere we currently have handling for timestamp with timezone we add in handling for timestamp without timezone with the same code except we gate it with a flag. #2282

…

Sent from my iPhone

On Mar 9, 2021, at 4:40 AM, Javier Sánchez Beltrán ***@***.***> wrote:  Hello @RussellSpitzer! I am having a similar issue and I agree to your solution as well. Did you continue working on this? I would like to know more about your approach. How would this work in practice? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.