[EPIC] Add support for Spark 4.0.1

[andygrove]

What is the problem the feature request solves?

edit: Updated to track remaining work for Spark 4.0.1 support.

Bugs / Incompatibility Issues:

• Reinstate macOS CI builds of Comet with Spark 4.0.0 #1949
• Broadcast hash join and sort-merge join can produce incorrect results on non-default collated keys [Spark 4] #4051
• [native_iceberg_compat] VariantShreddingSuite test failures with Spark 4.0.0 #2209

Testing

• Enable Iceberg tests with Spark 4
• Enable Spark SQL tests with native_iceberg_compat

New Features:

• Add support for MapSort expression in Spark 4.0.0 #1941

Already Fixed:

• Fix listagg-collation.sql test in Spark 4.0.0 #1947
• [EPIC] Fully support ANSI mode #313
• Feat: Support Spark 4.0.0 part1 #1830
• Remove COMET_ANSI_MODE_ENABLED config #1974
• Update 4.0.0.diff to reflect recent improvements in 3.5.5.diff #1846
• Add support for StringDecode in Spark 4.0.0 #1942
• Investigate failing tests with Spark 4.0.0 #1948
• Enable auto scan mode for Spark 4.0.0 #1967
• Fix Spark 4.0 sql tests #551
• [Spark 4.0.0] Can't zip RDDs with unequal numbers of partitions: List(9, 9, 9, 17) #2408
• Spark 4.0.x hive-1 jdk17 tests failing consistently in multiple PRs #2946
• Perform audit of all changes noted in https://spark.apache.org/docs/latest/sql-migration-guide.html#upgrading-from-spark-sql-35-to-40 and file issues as needed
• Update release scripts to publish Comet jars for Spark 4.0.0 #1989
• Add support for to_json compatibility with Spark 4.0.0. #3920
• CometSparkToColumnarExec missing from AQE plan on Spark 4 (two skipped tests) #4031

Describe the potential solution

No response

Additional context

No response

[dharanad]

ANSI Support EPIC #313

[kazuyukitanimura]

Investigate Guava duplicated classes #1968

[coderfender]

I can work on some of the low hanging issues @andygrove @kazuyukitanimura