[CORE] Reopen hudi GHA tests with Spark353

[Yohahaha]

Description

java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.catalog.SessionCatalog.invalidateCachedTable(Lorg/apache/spark/sql/catalyst/QualifiedTableName;)V
	at org.apache.spark.sql.hudi.command.DropHoodieTableCommand.run(DropHoodieTableCommand.scala:52)
	at org.apache.spark.sql.hudi.catalog.HoodieCatalog.dropTable(HoodieCatalog.scala:159)
	at org.apache.spark.sql.execution.datasources.v2.DropTableExec.run(DropTableExec.scala:38)

[zhouyuan]

Hi @Yohahaha,
The HUDI tests are not running with Spark-3.5 now, would you like to follow-up on this? If not we can help to re-enable this

-yuan

[Yohahaha]

@zhouyuan apache/hudi#12129 hudi may support v3.5.3 in next release version.

[zhztheplayer]

@Yohahaha I think we should keep hudi + Spark35 support in Gluten's next minor release (1.3.0). Do you think the timelines could match?

[Yohahaha]

@Yohahaha I think we should keep hudi + Spark35 support in Gluten's next minor release (1.3.0). Do you think the timelines could match?

it depends on when hudi and iceberg release supported version.

[zhztheplayer]

If it's uncertain, perhaps we need to revert the change and put off applying it? I am not sure how many of Spark35 users are also using hudi / iceberg. Maybe we can send out a brief discussion to more users?

cc @zzcclp @baibaichen

[Yohahaha]

If it's uncertain, perhaps we need to revert the change and put off applying it? I am not sure how many of Spark35 users are also using hudi / iceberg. Maybe we can send out a brief discussion to more users?

cc @zzcclp @baibaichen

Gluten only guarantee supports latest spark release version as talked with @zhouyuan before, gluten-datalake is downstream project compare to main gluten, I think it's hudi/iceberg's duty to keep compatibility with vanilla Spark, not Gluten.

[jackylee-ch]

Maybe we should discuss this on the development mailing list to get other people's opinions. In my opinion spark 3.5.3 is not a very good release and we have no intention to upgrade to 3.5.3 in our environment.

[baibaichen]

Hi All

Let's close this issue and move to #7637 for discuss.