[CORE] Iceberg and hudi module may breaks with Spark35

[Yohahaha]

Backend

VL (Velox)

Bug description

Gluten has upgrade Spark version to v3.5.3 #7537, but iceberg may has compatibility issue with Spark 3.5.3.
#7537 (comment)

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

No response

[philo-he]

Seems iceberg skips spark-3.5.3, see apache/iceberg#11160. cc @zhouyuan

[zhztheplayer]

@Yohahaha Similar to #7571, I think we should keep iceberg + Spark35 support in Gluten's next minor release (1.3.0). Any thoughts?

cc @baibaichen @zzcclp

[Yohahaha]

@Yohahaha Similar to #7571, I think we should keep iceberg + Spark35 support in Gluten's next minor release (1.3.0). Any thoughts?

iceberg decides skip spark 3.5.3, do we have better options?

[jackylee-ch]

apache/spark#48257 is the related spark pr that fix the extension problem for iceberg, and it will be released in spark 3.5.4. And iceberg would skip 3.5.3 and keep up with 3.5.4 after spark 3.5.4 is released.
I'm afraid they won't be released before we release Gluten 1.3.0.

[baibaichen]

@Yohahaha Similar to #7571, I think we should keep iceberg + Spark35 support in Gluten's next minor release (1.3.0). Any thoughts?

cc @baibaichen

@zhztheplayer @zhouyuan @zzcclp

It depends on which is impormant, supporting hudi and icerberg or supporting spark 3.5.3.

I guess spark 3.5.3 is not used in most cases, is there any reason that we must support spark 3.5.3? if not, I think revert to support 3.5.2 would be better

[zhztheplayer]

It seems #7537 didn't involve any code changes, it's probably easy to maintain as an individual patch.

@Yohahaha Would you exclude the possibility maintaining #7537 internally in your Gluten fork? Until iceberg / hudi bump their Spark version?

[Yohahaha]

@baibaichen @zhztheplayer any reason we must checkout release 1.3.0 from current main branch? why not checkout from the commit before 7537?
or we can checkout release 1.3.0 and revert 7537 on that branch.

The important things is Spark latest release version on branch 3.5 is v3.5.3, Gluten's strategy is keep latest Spark release version, do we want change it?

[baibaichen]

Gluten's strategy is keep latest Spark release version, do we want change it?

Gluten's strategy is supporting one version instead of latest release version due to lack of resource. Perhaps starting from version 3.5, we can support a stable version 3.5.2, and a latest version 3.5.3. Of course, this needs further disscussion.

Back to #7537, we don't change any meaningful codes, we can still using it in spark 3.5.3 even it was reverted.

[FelixYBW]

@Yohahaha Gluten tends to support latest spark release but since iceberg doesn't support spark3.5.3. Iceberg support is more important in Gluten now. Can we skip 3.5.3? once 3.5.4 fix the iceberg issue, we can add spark3.5.4.