HBASE-29203 There should be a StorefileSize equivalent to the TableSkewCost

[rmdmattingly]

I deployed this to a QA cluster at my company and enabled:

1. System table isolation (balancer conditional)
2. Replica distribution (balancer conditional)
3. and primarily balanced based on StoreFileSizeTableSkew and StoreFileSizeCost

We ended up with multiple tables, balanced virtually perfectly in every regard (the empty column is the host assigned to serving system tables):

I'm still thinking about how to unit test this. It is implicitly tested by our existing test suite since it has a default cost >0, but there is not much precedent for testing these cost functions individually and it would be non-trivial to meaningfully test it in isolation

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 29s		Docker mode activated.
-0 ⚠️	yetus	0m 2s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+1 💚	mvninstall	3m 3s		master passed
+1 💚	compile	0m 15s		master passed
+1 💚	javadoc	0m 12s		master passed
+1 💚	shadedjars	5m 55s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 6s		the patch passed
+1 💚	compile	0m 15s		the patch passed
+1 💚	javac	0m 15s		the patch passed
+1 💚	javadoc	0m 11s		the patch passed
+1 💚	shadedjars	6m 0s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	8m 21s		hbase-balancer in the patch passed.
		28m 41s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6825/7/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6825
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 4049bf6d1d47 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 3339597
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6825/7/testReport/
Max. process+thread count	270 (vs. ulimit of 30000)
modules	C: hbase-balancer U: hbase-balancer
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6825/7/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 45s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+1 💚	mvninstall	3m 54s		master passed
+1 💚	compile	0m 33s		master passed
+1 💚	checkstyle	0m 13s		master passed
+1 💚	spotbugs	0m 33s		master passed
+1 💚	spotless	0m 53s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 44s		the patch passed
+1 💚	compile	0m 27s		the patch passed
+1 💚	javac	0m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 9s		the patch passed
+1 💚	spotbugs	0m 37s		the patch passed
+1 💚	hadoopcheck	13m 45s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 52s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 11s		The patch does not generate ASF License warnings.
		34m 44s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6825/7/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6825
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux c6b1efeeb248 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 3339597
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	83 (vs. ulimit of 30000)
modules	C: hbase-balancer U: hbase-balancer
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6825/7/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

[ndimiduk]

Your pictures are too pretty to pass up. Let's learn what we can about how it performs at scale.

[ndimiduk]

Is this going to be too expensive for a cluster with 1000 servers and 100,000 regions?

This may be a legitimate use-case for re-implementing as a Stream with parallel processing enabled.

[rmdmattingly]

Luckily prepare isn't called too often, so hopefully it won't be too bad