fix: Support auto scan mode with Spark 4.0.0

[andygrove]

Which issue does this PR close?

Closes #1967

Rationale for this change

Provide better performance with Spark 4.0.0 by supporting auto scan mode. Also, increase test coverage of native_iceberg_compat, which we intend to eventually completely replace native_comet.

What changes are included in this PR?

• Remove auto fallback for Spark 4
• Fall back to Spark for scans with schemas containing collated string types
• Update Spark 4.0.0 diff to match changes made in other versions related to skipping tests for native scans
• Update Spark CollationSuite to expect Comet plans
• Ignore VariantShreddingSuite due to [native_iceberg_compat] VariantShreddingSuite test failures with Spark 4.0.0 #2209

How are these changes tested?

[codecov-commenter]

Codecov Report

❌ Patch coverage is 16.66667% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.53%. Comparing base (f09f8af) to head (df9d42f).
⚠️ Report is 410 commits behind head on main.

Files with missing lines	Patch %	Lines
...n/scala/org/apache/comet/rules/CometScanRule.scala	15.38%	6 Missing and 5 partials ⚠️
.../scala/org/apache/spark/sql/comet/util/Utils.scala	20.00%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1975      +/-   ##
============================================
+ Coverage     56.12%   58.53%   +2.40%     
- Complexity      976     1282     +306     
============================================
  Files           119      143      +24     
  Lines         11743    13254    +1511     
  Branches       2251     2364     +113     
============================================
+ Hits           6591     7758    +1167     
- Misses         4012     4264     +252     
- Partials       1140     1232      +92     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[andygrove]

hive-1 failure to investigate:

2025-07-09T20:18:09.4999985Z [info]   Cause: org.apache.comet.CometNativeException: External error: Arrow: Parquet argument error: Parquet error: encountered non UTF-8 data
2025-07-09T20:18:09.5000678Z [info]   at org.apache.comet.parquet.Native.readNextRecordBatch(Native Method)
2025-07-09T20:18:09.5001239Z [info]   at org.apache.comet.parquet.NativeBatchReader.loadNextBatch(NativeBatchReader.java:812)

[andygrove]

Current failures:

core-1:

$ grep "* FAIL" core1.txt 
2025-08-21T02:32:47.7183506Z [info] - SPARK-26677: negated null-safe equality comparison should not filter matched row groups *** FAILED *** (303 milliseconds)
2025-08-21T02:52:05.6178387Z [info] - scalar types rebuild *** FAILED *** (637 milliseconds)
2025-08-21T02:52:05.9761617Z [info] - object rebuild *** FAILED *** (357 milliseconds)
2025-08-21T02:52:06.3954010Z [info] - array rebuild *** FAILED *** (417 milliseconds)
2025-08-21T02:52:07.4434260Z [info] - malformed input *** FAILED *** (1 second, 40 milliseconds)
2025-08-21T02:52:07.6507720Z [info] - extract from shredded object *** FAILED *** (206 milliseconds)
2025-08-21T02:52:07.8370529Z [info] - extract from shredded array *** FAILED *** (182 milliseconds)
2025-08-21T02:52:08.0028967Z [info] - missing fields *** FAILED *** (166 milliseconds)
2025-08-21T02:52:08.1324041Z [info] - custom casts *** FAILED *** (129 milliseconds)

hive-1:

$ grep "* FAIL" hive1.txt 
2025-08-21T02:28:26.0691331Z [info] - SPARK-30201 HiveOutputWriter standardOI should use ObjectInspectorCopyOption.DEFAULT *** FAILED *** (375 milliseconds)

[comphead]

should we have a ticket?

[andygrove]

I removed this comment because Spark also does not track this information, as pointed out in #2190 (comment) by @parthchandra

[comphead]

isComplexType method now exists in in DataTypeSupport object

[andygrove]

Thanks. Updated.

[comphead]

lgtm thanks @andygrove