[SPARK-47457][SQL] Fix IsolatedClientLoader.supportsHadoopShadedClient to handle Hadoop 3.4+

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to fix IsolatedClientLoader.supportsHadoopShadedClient to handle Hadoop 3.4+ correctly.

Why are the changes needed?

Apache Spark 3.4+ support shaded clients, but currently supportsHadoopShadedClient returns false.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manual review.

Was this patch authored or co-authored using generative AI tooling?

No.

[LuciferYang]

+1, LGTM

[yaooqinn]

Hi @dongjoon-hyun. Can you fix the failed test?

[info] - SPARK-32212: test supportHadoopShadedClient() *** FAILED *** (3 milliseconds)
[info]   IsolatedClientLoader.supportsHadoopShadedClient(version) was true version 4 (HadoopVersionInfoSuite.scala:81)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
[info]   at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
[info]   at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
[info]   at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
[info]   at org.apache.spark.sql.hive.client.HadoopVersionInfoSuite.$anonfun$new$8(HadoopVersionInfoSuite.scala:81)

[dongjoon-hyun]

Thank you, @yaooqinn and @LuciferYang . I fixed the last failure.

$ build/sbt "hive/testOnly *.HadoopVersionInfoSuite -z SPARK-32212" -Phive
...
[info] - SPARK-32256: Hadoop VersionInfo should be preloaded (36 seconds, 981 milliseconds)
[info] - SPARK-32212: test supportHadoopShadedClient() (2 milliseconds)
[info] - SPARK-32212: built-in Hadoop version should support shaded client (1 millisecond)
...
[info] Run completed in 37 seconds, 730 milliseconds.
[info] Total number of tests run: 3
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 3, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 49 s, completed Mar 19, 2024, 7:47:45 AM

[dongjoon-hyun]

Merged to master for Apache Spark 4.0.0.