[EPIC] [DISCUSS] Comet timezone handling

[andygrove]

What is the problem the feature request solves?

Comet currently assumes that all native processing uses the UTC timezone. When reading from Parquet sources, Comet converts timestamps to UTC.

      String timeZoneId = conf.get("spark.sql.session.timeZone");
      // Native code uses "UTC" always as the timeZoneId when converting from spark to arrow schema.
      Schema arrowSchema = Utils$.MODULE$.toArrowSchema(sparkSchema, "UTC");
      byte[] serializedRequestedArrowSchema = serializeArrowSchema(arrowSchema);
      Schema dataArrowSchema = Utils$.MODULE$.toArrowSchema(dataSchema, "UTC");
      byte[] serializedDataArrowSchema = serializeArrowSchema(dataArrowSchema);

However, we are now seeing that this causes correctness issues or exceptions when the data source is not Parquet:

• Spark SQL test failures due to timestamp mismatch when LocalTableScan is native #2720
• date_trunc incorrect results in non-UTC timezone #2649

This epic is for reviewing and discussing Comet's approach to time zones.

Describe the potential solution

No response

Additional context

No response

[parthchandra]

when the data source is not Parquet:

when is this true?

[andygrove]

when the data source is not Parquet:

when is this true?

For Comet sinks, such as:

• LocalTableScanExec
• CometSparkToColumnarExec
• Exchanges
• UnionExec
• CoalesceExec

[andygrove]

Another related timezone issue:

Cast from string to timestamp:

      case DataTypes.TimestampType if timeZoneId.exists(tz => tz != "UTC") =>
        Incompatible(Some(s"Cast will use UTC instead of $timeZoneId"))