[EPIC] Iceberg compatibility fixes

[andygrove]

What is the problem the feature request solves?

The integration between Comet and Iceberg is not fully working. This epic is to track the work that we are targeting for the Comet 0.10.0 release to improve the integration.

• [iceberg] Comet cannot execute some iceberg metadata table queries #2033
• Use vanilla Spark in Iceberg integration tests in CI #1986
• [iceberg] Error loading in-memory sorter check class path #1982
• [iceberg] testCopyOnWriteMergeWithoutShufflesWithPredicate failure (MutableBuffer LayoutError) #2087
• [iceberg] CometBypassMergeSortShuffleWriter freePage error #2088
• [iceberg] Document configuration flags needed for Comet Iceberg to work correctly #2092
• [iceberg] Query plan doesn't match when Comet scan is enabled #2120
• [iceberg] SIGSEGV while testing Storage Partition Join #2121
• [iceberg] Memory corruption due to unsafe reuse of mutable buffers #2131

Related to broadcast exchange:

• [iceberg] Iceberg data files not found when Comet vectorized reader is enabled #2116

Related to shuffle exchange:

• [iceberg] testMergeOnReadMerge failure creating empty StructArray #2086

Related to deleted rows:

• [iceberg] RecordBatch might have logical row mapping on physical arrays #974
• [iceberg] Storage Partition Join (SPJ) returns mismatch results #2119
• [iceberg] TestSparkDataWrite show mismatch results #2118
• [iceberg] Comet execution only takes Arrow Arrays, but got class org.apache.iceberg.spark.data.vectorized.ColumnVectorWithFilter #2117

Tracking PRs for Iceberg changes:

• [iceberg] Tracking PR to update Iceberg for Parquet shading issues #2093
• [iceberg] Tracking PR to update Iceberg to enable Comet native execution with Iceberg #2094
• [iceberg] Tracking PR to update useDecimal128 in Iceberg #2095
• [iceberg] Tracking PR for deleted rows support #2390

Nice to have:

• Update fuzz testing tool to support testing with Comet + Iceberg integration #1984

Describe the potential solution

No response

Additional context

No response