HBASE-29459 Capture bulkload files only till IncrCommittedWalTs during Incremental Backup #7166

ankitsol · 2025-07-21T18:31:44Z

In incremental backup, we copy bulkload files to backup location (along with Hfiles generated from WALs). We introduced IncrCommittedWalTs field for incremental backup and this needs to be used in copy bulkload code so only bulkload files till IncrCommittedWalTs timestamp are copied to incremental backup

JIRA: https://issues.apache.org/jira/browse/HBASE-29459

kgeisz

LGTM

taklwu

just a question to understand the logic, but otherwise LGTM

taklwu · 2025-07-22T00:31:01Z

hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupSystemTable.java

          }
-          result.add(new BulkLoad(table, region, fam, path, row));
-          LOG.debug("found orig " + path + " for " + fam + " of table " + region);
+          LOG.debug("found orig {} for {} of table {} with timestamp {}", path, fam, region,


nit adjust the short form to clear wording

Suggested change

LOG.debug("found orig {} for {} of table {} with timestamp {}", path, fam, region,

LOG.debug("found original path {} for column family {} of table {} with timestamp {}", path, fam, region,

taklwu · 2025-07-22T00:36:06Z

hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupSystemTable.java

+          long timestamp = 0L;
          for (Cell cell : res.listCells()) {
            row = CellUtil.cloneRow(cell);
+            timestamp = cell.getTimestamp();


so is this timestamp the IncrCommittedWalTs? I cannot find other timestamp representing it but it looks like this PR share the same setup as #7150 that used the BulkLoad#timestamp as IncrCommittedWalTs?

This timestamp is Bulkload timestamp (time at which Bulkload operation is triggered). This will be saved in backup system table and compared with IncrCommittedWalTs during incremental backup

please fix spotless, spotbugs, javac and blanks

taklwu · 2025-07-22T04:09:39Z

...e-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java

    List<String> activeFiles = new ArrayList<>();
    List<String> archiveFiles = new ArrayList<>();
-    List<BulkLoad> bulkLoads = backupManager.readBulkloadRows(tablesToBackup);
+    List<BulkLoad> bulkLoads = new ArrayList<>();


Suggested change

List<BulkLoad> bulkLoads = new ArrayList<>();

List<BulkLoad> bulkLoads = backupInfo.isContinuousBackupEnabled() ?

backupManager.readBulkloadRows(tablesToBackup, backupInfo.getIncrCommittedWalTs()) :

backupManager.readBulkloadRows(tablesToBackup);

this may help the spotbugs error

Apache-HBase · 2025-07-22T08:32:20Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 36s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ HBASE-28957 Compile Tests _
+1 💚	mvninstall	3m 19s		HBASE-28957 passed
+1 💚	compile	0m 33s		HBASE-28957 passed
-0 ⚠️	checkstyle	0m 10s	/buildtool-branch-checkstyle-hbase-backup.txt	The patch fails to run checkstyle in hbase-backup
+1 💚	spotbugs	0m 32s		HBASE-28957 passed
+1 💚	spotless	0m 48s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 6s		the patch passed
+1 💚	compile	0m 31s		the patch passed
+1 💚	javac	0m 31s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 9s	/buildtool-patch-checkstyle-hbase-backup.txt	The patch fails to run checkstyle in hbase-backup
+1 💚	spotbugs	0m 37s		the patch passed
+1 💚	hadoopcheck	12m 20s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 45s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 10s		The patch does not generate ASF License warnings.
		31m 23s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7166/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7166
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux d5db18a684cb 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-28957 / `674074b`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	83 (vs. ulimit of 30000)
modules	C: hbase-backup U: hbase-backup
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7166/3/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-07-22T08:45:06Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 40s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ HBASE-28957 Compile Tests _
+1 💚	mvninstall	3m 15s		HBASE-28957 passed
+1 💚	compile	0m 19s		HBASE-28957 passed
+1 💚	javadoc	0m 14s		HBASE-28957 passed
+1 💚	shadedjars	5m 56s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 5s		the patch passed
+1 💚	compile	0m 19s		the patch passed
+1 💚	javac	0m 19s		the patch passed
+1 💚	javadoc	0m 13s		the patch passed
+1 💚	shadedjars	5m 54s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
-1 ❌	unit	23m 16s	/patch-unit-hbase-backup.txt	hbase-backup in the patch failed.
		44m 11s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7166/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7166
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux c2d3b1bd6bce 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-28957 / `674074b`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7166/3/testReport/
Max. process+thread count	3751 (vs. ulimit of 30000)
modules	C: hbase-backup U: hbase-backup
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7166/3/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

…g Incremental Backup (#7166) Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org> Reviewed by: Kevin Geiszler <kevin.j.geiszler@gmail.com>

…g Incremental Backup (apache#7166) Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org> Reviewed by: Kevin Geiszler <kevin.j.geiszler@gmail.com>

…g Incremental Backup (#7166) Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org> Reviewed by: Kevin Geiszler <kevin.j.geiszler@gmail.com>

asolomonC added 2 commits July 21, 2025 23:56

HBASE-29459 Capture bulkload files only till IncrCommittedWalTs

e668110

Minor test fix

12ef85a

This comment has been minimized.

Sign in to view

kgeisz approved these changes Jul 22, 2025

View reviewed changes

taklwu reviewed Jul 22, 2025

View reviewed changes

taklwu previously approved these changes Jul 22, 2025

View reviewed changes

taklwu reviewed Jul 22, 2025

View reviewed changes

Spotless fix

674074b

ankitsol requested a review from taklwu July 22, 2025 15:01

taklwu approved these changes Jul 22, 2025

View reviewed changes

taklwu merged commit 15dc75f into apache:HBASE-28957 Jul 22, 2025
1 check failed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HBASE-29459 Capture bulkload files only till IncrCommittedWalTs during Incremental Backup #7166

HBASE-29459 Capture bulkload files only till IncrCommittedWalTs during Incremental Backup #7166

Uh oh!

ankitsol commented Jul 21, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

kgeisz left a comment

Uh oh!

taklwu left a comment

Uh oh!

taklwu Jul 22, 2025

Uh oh!

taklwu Jul 22, 2025

Uh oh!

ankitsol Jul 22, 2025

Uh oh!

taklwu Jul 22, 2025 •

edited

Loading

Uh oh!

taklwu Jul 22, 2025

Uh oh!

Apache-HBase commented Jul 22, 2025

Uh oh!

Apache-HBase commented Jul 22, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

	LOG.debug("found orig {} for {} of table {} with timestamp {}", path, fam, region,
	LOG.debug("found original path {} for column family {} of table {} with timestamp {}", path, fam, region,

HBASE-29459 Capture bulkload files only till IncrCommittedWalTs during Incremental Backup #7166

HBASE-29459 Capture bulkload files only till IncrCommittedWalTs during Incremental Backup #7166

Uh oh!

Conversation

ankitsol commented Jul 21, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

kgeisz left a comment

Choose a reason for hiding this comment

Uh oh!

taklwu left a comment

Choose a reason for hiding this comment

Uh oh!

taklwu Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

taklwu Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

ankitsol Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

taklwu Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

taklwu Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

Apache-HBase commented Jul 22, 2025

Uh oh!

Apache-HBase commented Jul 22, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

taklwu Jul 22, 2025 •

edited

Loading