HBASE-27126 Support multi-threads cleaner for MOB files #5833

chandrasekhar-188k · 2024-04-17T15:32:33Z

No description provided.

hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCleanerChore.java

Apache9 · 2024-05-29T15:15:26Z

Please use shaded version of guava classes?

Thanks.

pankaj72981 · 2024-06-03T03:35:47Z

hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCleanerChore.java

 public class MobFileCleanerChore extends ScheduledChore {

  private static final Logger LOG = LoggerFactory.getLogger(MobFileCleanerChore.class);
+  public static final int TIME_OUT_VALUE = 30;


Please make it configurable and 30 minutes is too large value.

added a configuration for it. default it set at 5 mins

pankaj72981 · 2024-06-03T03:39:45Z

hbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestExpiredMobFileCleanerChore.java

+  @BeforeClass
+  public static void setUpBeforeClass() throws Exception {
+    TEST_UTIL.getConfiguration().setInt("hfile.format.version", 3);
+    TEST_UTIL.getConfiguration().setInt(MOB_CLEANER_BATCH_SIZE_UPPER_BOUND, 2);


Shall we add UT for MOB_CLEANER_THREAD_COUNT > 1 also?

added test case for multi thread and single thread scenarios

Apache9 · 2025-08-31T12:46:30Z

Any updates here? @chandrasekhar-188k

Thanks.

chandrasekhar-188k · 2025-09-02T02:09:26Z

Any updates here? @chandrasekhar-188k

Thanks.

I have fixed the review comments and rebased the PR

chandrasekhar-188k · 2025-09-02T02:10:08Z

@pankaj72981
review comments are adressed, pls help to review again.

chandrasekhar-188k · 2025-09-20T03:31:47Z

@chandrasekhar-188k Any updates here?
Thanks.

I am working on the review comment fixes, i will update the PR once its finished

review comments are fixed, pls check the changes

Apache9 · 2025-10-07T12:26:23Z

hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCleanerChore.java

-          MobFileCleanupUtil.cleanupObsoleteMobFiles(master.getConfiguration(), htd.getTableName(),
-            admin);
+        future.get(cleanerFutureTimeout, TimeUnit.SECONDS);
+      } catch (InterruptedException | ExecutionException | TimeoutException e) {


If the current thread is interrupted, future.get will return immediately with an InterruptedException...

The InterruptedException here is for interrupting the current thread, not the task in the thread pool.

Apache9 · 2025-10-07T12:27:53Z

hbase-server/src/test/java/org/apache/hadoop/hbase/mob/TestExpiredMobFileCleanerChore.java

+   * run.
+   */
+
+  public static void testCleanerInternal() throws Exception {


Why make this method public?

updated to private

hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCleanerChore.java

Apache-HBase · 2025-10-10T04:08:57Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 36s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+1 💚	mvninstall	3m 37s		master passed
+1 💚	compile	3m 29s		master passed
+1 💚	checkstyle	0m 40s		master passed
+1 💚	spotbugs	1m 40s		master passed
+1 💚	spotless	0m 48s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 0s		the patch passed
+1 💚	compile	3m 28s		the patch passed
+1 💚	javac	3m 28s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 40s		the patch passed
+1 💚	spotbugs	1m 43s		the patch passed
+1 💚	hadoopcheck	11m 57s		Patch does not cause any errors with Hadoop 3.3.6 3.4.1.
+1 💚	spotless	0m 46s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 11s		The patch does not generate ASF License warnings.
		40m 17s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5833/17/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5833
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux d803626f91c2 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `1c09b52`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	85 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5833/17/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-10-10T07:26:13Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 29s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+1 💚	mvninstall	3m 25s		master passed
+1 💚	compile	0m 58s		master passed
+1 💚	javadoc	0m 28s		master passed
+1 💚	shadedjars	6m 3s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 2s		the patch passed
+1 💚	compile	0m 56s		the patch passed
+1 💚	javac	0m 56s		the patch passed
+1 💚	javadoc	0m 27s		the patch passed
+1 💚	shadedjars	6m 2s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	210m 55s		hbase-server in the patch passed.
		237m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5833/17/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#5833
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 2baba6962b7f 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `1c09b52`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5833/17/testReport/
Max. process+thread count	4180 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5833/17/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Pankaj Kumar <pankajkumar@apache.org>

Signed-off-by: Peng Lu <lupeng@apache.org> (cherry picked from commit f800a13)

This comment has been minimized.

Sign in to view

Apache9 reviewed Apr 20, 2024

View reviewed changes

hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCleanerChore.java Outdated Show resolved Hide resolved

chandrasekhar-188k force-pushed the br_HBASE-27126 branch from 7716f46 to f269274 Compare April 26, 2024 08:16

This comment has been minimized.

Sign in to view

pankaj72981 reviewed Jun 3, 2024

View reviewed changes

chandrasekhar-188k force-pushed the br_HBASE-27126 branch from f269274 to b482a58 Compare June 8, 2024 15:19

This comment has been minimized.

Sign in to view

chandrasekhar-188k force-pushed the br_HBASE-27126 branch from b482a58 to 8a7bd16 Compare September 25, 2024 16:55

This comment has been minimized.

Sign in to view

chandrasekhar-188k force-pushed the br_HBASE-27126 branch from 8a7bd16 to b27697b Compare September 2, 2025 02:08

chandrasekhar-188k requested a review from pankaj72981 September 2, 2025 02:09

This comment has been minimized.

Sign in to view

chandrasekhar-188k requested a review from Apache9 September 20, 2025 03:28

This comment has been minimized.

Sign in to view

chandrasekhar-188k and others added 3 commits September 20, 2025 11:46

HBASE-27126 Support multi-threads cleaner for MOB files

c2ce6c7

Fix UT Failures

c5521ef

Fix UT Failures

3be4cfb

chandrasekhar-188k force-pushed the br_HBASE-27126 branch from dc05a56 to 3ae9e1f Compare September 20, 2025 06:18

Fix Review comments

ed1a1dd

chandrasekhar-188k force-pushed the br_HBASE-27126 branch from 3ae9e1f to ed1a1dd Compare September 20, 2025 06:21

This comment has been minimized.

Sign in to view

Apache9 reviewed Oct 7, 2025

View reviewed changes

Fix Review comments - New

a499cb4

chandrasekhar-188k force-pushed the br_HBASE-27126 branch from 13a864e to a499cb4 Compare October 8, 2025 03:05

This comment has been minimized.

Sign in to view

Apache9 approved these changes Oct 8, 2025

View reviewed changes

pankaj72981 reviewed Oct 9, 2025

View reviewed changes

hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCleanerChore.java Outdated Show resolved Hide resolved

hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobFileCleanerChore.java Outdated Show resolved Hide resolved

Fix Review comments - 2

1c09b52

pankaj72981 approved these changes Oct 25, 2025

View reviewed changes

chandrasekhar-188k merged commit f800a13 into apache:master Nov 1, 2025
1 check passed

chandrasekhar-188k added a commit that referenced this pull request Nov 3, 2025

HBASE-27126 Support multi-threads cleaner for MOB files (#5833)

7c1c78d

Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Pankaj Kumar <pankajkumar@apache.org>

chandrasekhar-188k added a commit that referenced this pull request Dec 13, 2025

HBASE-27126 Support multi-threads cleaner for MOB files (#7430)(#5833)

163a1eb

Signed-off-by: Peng Lu <lupeng@apache.org> (cherry picked from commit f800a13)

chandrasekhar-188k added a commit that referenced this pull request Dec 13, 2025

HBASE-27126 Support multi-threads cleaner for MOB files (#7430)(#5833)

6ab164c

Signed-off-by: Peng Lu <lupeng@apache.org> (cherry picked from commit f800a13)

chandrasekhar-188k added a commit that referenced this pull request Dec 13, 2025

HBASE-27126 Support multi-threads cleaner for MOB files (#7430)(#5833)

bc8c875

Signed-off-by: Peng Lu <lupeng@apache.org> (cherry picked from commit f800a13)

HBASE-27126 Support multi-threads cleaner for MOB files #5833

HBASE-27126 Support multi-threads cleaner for MOB files #5833

Uh oh!

Conversation

chandrasekhar-188k commented Apr 17, 2024

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Apache9 commented May 29, 2024

Uh oh!

pankaj72981 Jun 3, 2024

Choose a reason for hiding this comment

Uh oh!

chandrasekhar-188k Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

pankaj72981 Jun 3, 2024

Choose a reason for hiding this comment

Uh oh!

chandrasekhar-188k Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Apache9 commented Aug 31, 2025

Uh oh!

chandrasekhar-188k commented Sep 2, 2025

Uh oh!

chandrasekhar-188k commented Sep 2, 2025

Uh oh!

This comment has been minimized.

chandrasekhar-188k commented Sep 20, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Apache9 Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

Apache9 Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

chandrasekhar-188k Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

Uh oh!

Apache-HBase commented Oct 10, 2025

Uh oh!

Apache-HBase commented Oct 10, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants