[Spark 4.0] Add string collation support

[andygrove]

What is the problem the feature request solves?

A major new feature of Spark 4.0.0 is string collation support. Unfortunately, DataFusion does not support string collation (apache/datafusion#9192); therefore, Comet does not support it either.

Describe the potential solution

In the short term, we likely need to fall back to Spark for any queries involving string collation.

Additional context

https://www.databricks.com/blog/introducing-collations-databricks

https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-collation

[parthchandra]

I don't understand what this needs from native_iceberg_compat. Collation sequences are needed for sort. Is the requirement to ensure that the Spark type does not drop the collation information during the scan?

[andygrove]

I don't understand what this needs from native_iceberg_compat. Collation sequences are needed for sort. Is the requirement to ensure that the Spark type does not drop the collation information during the scan?

I really don't know yet. For now, we just want to fall back to Spark for any queries where the schema has string collation types and the scan is the first place where we see this.

It has been a while since I started looking into this, but IIRC, I was getting errors in the scan when converting the Spark schema to the Arrow schema because we did not recognize the string collation types.

[parthchandra]

converting the Spark schema to the Arrow schema because we did not recognize the string collation types

Hmm. Spark Schema -> Arrow Schema -> Spark Schema is going to drop the collation type even if we address this in the conversion method.
Spark's conversion methods don't worry about it - https://github.com/apache/spark/blob/c863717831029c44069ec17dcf8091147e14a7b3/sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala#L41