[SPARK-51164][CORE][TESTS][FOLLOWUP] Add hadoop.caller.context.enabled for scalatest-maven-plugin.

[cnauroth]

What changes were proposed in this pull request?

Apply previous SPARK-51164 fix to mvn test invocations of scalatest-maven-plugin.

Why are the changes needed?

I found that the previous SPARK-51164 fix was working well for sbt test invocations. However, I was seeing UtilsSuite fail when invoked through mvn. The previous fix applied the hadoop.caller.context.enabled configuration to the Java tests run through maven-surefire-plugin. This patch applies the same change for the Scala tests run through scalatest-maven-plugin.

Does this PR introduce any user-facing change?

No. This change only impacts tests.

How was this patch tested?

build/mvn -o -pl core test -Dtest=none -Dsuites=org.apache.spark.util.UtilsSuite

Run completed in 8 seconds, 900 milliseconds.
Total number of tests run: 61
Suites: completed 1, aborted 0
Tests: succeeded 61, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

Got it. So, previously, I added this to maven-surefire-plugin only and missed scalatest-maven-plugin. Thank you, @cnauroth !

[dongjoon-hyun]

+1, LGTM.

[dongjoon-hyun]

Since the PR builder is SBT, I verified manually.

$ build/mvn -pl core test -Dtest=none -Dsuites=org.apache.spark.util.UtilsSuite
...
UtilsSuite:
- timeConversion
- Test byteString conversion
- bytesToString
- copyStream
- copyStreamUpTo
- memoryStringToMb
- splitCommandString
- string formatting of time durations
- reading offset bytes of a file
- reading offset bytes of a file (compressed)
- reading offset bytes across multiple files
- reading offset bytes across multiple files (compressed)
- deserialize long value
- writeByteBuffer should not change ByteBuffer position
- get iterator size
- getIteratorZipWithIndex
- SPARK-35907: createDirectory
- doesDirectoryContainFilesNewerThan
- resolveURI
- resolveURIs with multiple paths
- nonLocalPaths
- isBindCollision
- log4j log level change
- deleteRecursively
- SPARK-50716: deleteRecursively - SymbolicLink To File
- SPARK-50716: deleteRecursively - SymbolicLink To Dir
- loading properties from file
- timeIt with prepare
- fetch hcfs dir
- shutdown hook manager
- isInDirectory
- circular buffer: if nothing was written to the buffer, display nothing
- circular buffer: if the buffer isn't full, print only the contents written
- circular buffer: data written == size of the buffer
- circular buffer: multiple overflow
- isDynamicAllocationEnabled
- getDynamicAllocationInitialExecutors
- Set Spark CallerContext
- encodeFileNameToURIRawPath
- decodeFileNameInURI
- Kill process
- chi square test of randomizeInPlace
- redact sensitive information
- redact sensitive information in command line args
- redact sensitive information in sequence of key value pairs
- tryWithSafeFinally
- tryWithSafeFinallyAndFailureCallbacks
- load extensions
- check Kubernetes master URL
- stringHalfWidth
- trimExceptCRLF standalone
- pathsToMetadata
- checkHost supports both IPV4 and IPV6
- checkHostPort support IPV6 and IPV4
- parseHostPort support IPV6 and IPV4
- executorOffHeapMemorySizeAsMb when MEMORY_OFFHEAP_ENABLED is false
- executorOffHeapMemorySizeAsMb when MEMORY_OFFHEAP_ENABLED is true
- executorMemoryOverhead when MEMORY_OFFHEAP_ENABLED is true, but MEMORY_OFFHEAP_SIZE not config scene
- isPushBasedShuffleEnabled when PUSH_BASED_SHUFFLE_ENABLED and SHUFFLE_SERVICE_ENABLED are both set to true in YARN mode with maxAttempts set to 1
- doTryWithCallerStacktrace and getTryWithCallerStacktrace
- nested doTryWithCallerStacktrace and getTryWithCallerStacktrace
Run completed in 2 seconds, 936 milliseconds.
Total number of tests run: 61
Suites: completed 1, aborted 0
Tests: succeeded 61, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:17 min
[INFO] Finished at: 2025-02-11T17:27:24-08:00
[INFO] ------------------------------------------------------------------------

[dongjoon-hyun]

Merged to master/4.0/3.5.

[dongjoon-hyun]

Thank you again, @cnauroth , and sorry for the trouble on Maven testing. :)

[cnauroth]

@dongjoon-hyun , thanks for the quick approval and merge. No worries!