[VL] Spark 4.1: Support checksum-based shuffle writers (SPARK-53322)

[baibaichen]

Backend

VL (Velox)

Bug description

Spark 4.1 introduced checksum-based shuffle writers (SPARK-53322/SPARK-54439, sub-tasks of SPARK-51166 Umbrella). Gluten's shuffle writer implementation does not produce the expected checksums, causing test failures.

Spark 4.1 only.

Parent issue: #11550

Impact

Suite	Status	spark40	spark41
GlutenMapStatusEndToEndSuite	Entire suite TODO	🟢	🔴
GlutenKeyGroupedPartitioningSuite	.excludeByPrefix("SPARK-53322")	🟢	🔴
GlutenKeyGroupedPartitioningSuite	.excludeByPrefix("SPARK-54439")	🟢	🔴

References

• Apache Spark JIRA: SPARK-53322
• Apache Spark JIRA: SPARK-54439
• Apache Spark JIRA: SPARK-51166 (Umbrella)