HDFS-16740 backport mini cluster flakiness fixes to 3.3

[mccormickt12]

Description of PR

backport mini cluster flakiness fixes to 3.3

How was this patch tested?

Testing needs to be done via github checks to ensure they pass / improve

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[mccormickt12]

These 3 files had conflicts, they all seemed pretty simple to resolve (all related to imports)

        both modified:   hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/AdminStatesBaseTest.java
        both modified:   hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
        both modified:   hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	4m 4s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 13 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	33m 23s		branch-3.3 passed
+1 💚	compile	0m 47s		branch-3.3 passed
+1 💚	checkstyle	0m 31s		branch-3.3 passed
+1 💚	mvnsite	0m 54s		branch-3.3 passed
+1 💚	javadoc	1m 6s		branch-3.3 passed
+1 💚	spotbugs	1m 52s		branch-3.3 passed
+1 💚	shadedclient	22m 18s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
-1 ❌	mvninstall	0m 30s	/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 ❌	compile	0m 32s	/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 ❌	javac	0m 32s	/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 18s	/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	The patch fails to run checkstyle in hadoop-hdfs
-1 ❌	mvnsite	0m 33s	/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
+1 💚	javadoc	0m 56s		the patch passed
-1 ❌	spotbugs	0m 31s	/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
-1 ❌	shadedclient	8m 31s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	0m 32s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
+1 💚	asflicense	0m 22s		The patch does not generate ASF License warnings.
		76m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6325/1/artifact/out/Dockerfile
GITHUB PR	#6325
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux abbbfcae4b8d 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / 4536205
Default Java	Private Build-1.8.0_362-8u372-gaus1-0ubuntu118.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6325/1/testReport/
Max. process+thread count	565 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6325/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	9m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 13 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	51m 0s		branch-3.3 passed
+1 💚	compile	1m 18s		branch-3.3 passed
+1 💚	checkstyle	0m 55s		branch-3.3 passed
+1 💚	mvnsite	1m 27s		branch-3.3 passed
+1 💚	javadoc	1m 34s		branch-3.3 passed
+1 💚	spotbugs	3m 22s		branch-3.3 passed
+1 💚	shadedclient	42m 48s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 17s		the patch passed
+1 💚	compile	1m 10s		the patch passed
+1 💚	javac	1m 10s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 44s		hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 263 unchanged - 1 fixed = 263 total (was 264)
+1 💚	mvnsite	1m 16s		the patch passed
+1 💚	javadoc	1m 24s		the patch passed
+1 💚	spotbugs	3m 18s		the patch passed
+1 💚	shadedclient	41m 48s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	224m 55s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 42s		The patch does not generate ASF License warnings.
		388m 33s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestRollingUpgrade
	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
	hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6325/2/artifact/out/Dockerfile
GITHUB PR	#6325
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 6a30b69f65e0 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / 0d6eee4
Default Java	Private Build-1.8.0_362-8u372-gaus1-0ubuntu118.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6325/2/testReport/
Max. process+thread count	2272 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6325/2/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[mccormickt12]

@simbadzina @goiri Only a couple tests failed after this backport, which seems a lot better than previously. Not sure if the expectation was everything passes.
cc @steveloughran in case you have more context

[steveloughran]

+1

[steveloughran]

merged, I just managed to get the commit message slightly wrong (after 4 weeks off I can't type any more), so rolled back, edited the message and resubmitted