HDDS-13912. Modularise Snapshot Delta file computer (Full Diff) #9283

swamirishi · 2025-11-11T23:43:53Z

What changes were proposed in this pull request?

Modularise SnapshotDiff computation for figuring out delta files between 2 snapshots.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13912

How was this patch tested?

Added unit tests

Change-Id: Iba47aeb21663dfa407ab71339cef02c0d74b49f2

Change-Id: Ifd2feca1fddb144e4955db025f0b15a2ab1f3bfe

Change-Id: I34536ff06efb7d5a4942853f0fd83942ab398b5f

…otLocalDataManager Change-Id: I34536ff06efb7d5a4942853f0fd83942ab398b5f

Change-Id: I32bcaf2a1fb290f1790c02872a0230cd65586636

Change-Id: I105a2e8178c0444d52de41b99801f4ceb6d57ffd # Conflicts: # hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/Checksum.java # hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/ObjectSerializer.java # hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/YamlSerializer.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalDataYaml.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotLocalDataManager.java # hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotLocalDataYaml.java

Change-Id: I985170e38fb8beeb784048e85a08a4c79e1aec97

Change-Id: I33e6e6e825bf23c323ad7ed593d800a11720fa4f

Change-Id: If30b2c766db82adde72145c8ecd3e590ef54cc2d

Change-Id: Id3f2c49050bc3476b9e0f5f51dacb6d9acc4c2f7

Change-Id: I432960725b4c6c55aa906b5780cc3027e41e10db

Change-Id: I3c5514e5bbd251a2b5297d8f074cfde5c71fa543

Change-Id: Ib5a9e6c91bdccba17820263c47eaf2c8400e930d

Change-Id: Ica36e0615c7bc6aa9b6a7f6fafafd0f830d4bafb

Change-Id: I26b66f266bb7677e4b1078f5fcd9f2ce3a651a70 # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotLocalDataManager.java

Change-Id: I1d93dbc048a42cc55ff1f8ffa420e52f967527b8

Change-Id: I34202928a7a367dd0a1e57219317ff34de352b78

Change-Id: Iad6f26cb71ec921c51ee2d138745df1a2663533f

Change-Id: Ic5f7e249cfb9cb3973cbcd4abd36b22a6ff8f5aa

…calDataProvider Change-Id: I3a004b4b435075a4348960aeed642e8da71e7e72

Change-Id: I06990bc9ab8fc7e1eb7bec255646a650bd8c35fe

Change-Id: I4c6c61c83aa9fadab8ecef854b99dcc0a89a2208

Change-Id: I0e476322372a302572f1fe79cbf2e874bfeac2ed

Change-Id: I31004e0c95dad64411c6fe848501a82f2f773cba

Change-Id: Id317c8b56e8b25c122b68eaf96599b9690d08f79

Change-Id: I3849387d064e093634e69cdaf870d27c1934cda5 # Conflicts: # hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/ObjectSerializer.java # hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/util/YamlSerializer.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalData.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalDataYaml.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotLocalDataManager.java

Change-Id: Ie5e5f3dab4324103e8855dd15619d7755f0422e6

Change-Id: I55bd5c3ef7fc32910a9111328638de2edffcd541

Change-Id: I880997d3eebdf378f14c203c61c2d63b2d17552e

Change-Id: I13ba8e2fd012a3c964d657e83496c93a4f55a3be

Change-Id: Iaad718f0c7bc23c67650144c0c7ac26f0060a437

Change-Id: Id2be646eeb0aa1d264211bf2950803bcb2839ce9

Change-Id: I83fe0a362530c9aeddee5f959b135a7b282e617f # Conflicts: # hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/StringInMemoryTestTable.java # hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdiff/RocksDBCheckpointDiffer.java

Change-Id: Ie3d199a6401b01af35e63781e2b6656d6aa1d5e9

Change-Id: I1d7aded575ac6daaa5f78b3be9f1821d94cb24c6

Change-Id: Iba0dd30952d17d11eae900213437aa4c28564448

Change-Id: Idba640c859ed4906842f9fafebf04825409053ef

Change-Id: I47fca4c2ca1738b533992851fa54291cb109cc39

Change-Id: Id07b78b650ce58fcc71bbac478d085b139123b02

Change-Id: Iacf5dd1fd2013d6203ea7003645cb79433186519

Change-Id: I22a212c84a3dbcd66a8e61a940d4df239b73133f

Change-Id: Idc5174ef66392fe16693aa8799aa51db7277a0e0 # Conflicts: # hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/RdbUtil.java # hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/SstFileInfo.java # hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdiff/RocksDBCheckpointDiffer.java # hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdiff/RocksDiffUtils.java

Change-Id: Ib313080b2ffa2f6e7545ae817f5e456cabc5dba7

Copilot

Pull Request Overview

This PR modularizes the snapshot delta file computation logic by introducing a clean abstraction layer. The changes extract the getINode utility method to a common location and create a new package structure with an interface-based design for computing delta files between snapshots.

Introduces DeltaFileComputer interface as a contract for delta file computation
Creates FileLinkDeltaFileComputer abstract base class that handles hard link creation and common operations
Implements FullDiffComputer as a concrete implementation using inode-based comparison with fallback to full file list comparison
Moves getINode utility from OmSnapshotUtils to IOUtils for broader reuse across modules

Reviewed Changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 16 comments.

Show a summary per file

File	Description
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/DeltaFileComputer.java	Defines the public interface for delta file computation between snapshots
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/FileLinkDeltaFileComputer.java	Abstract base class implementing common hard link creation logic and snapshot management
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/FullDiffComputer.java	Package-private concrete implementation performing full diff computation with inode optimization
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/package-info.java	Package documentation for the new delta computation package
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/IOUtils.java	Adds relocated `getINode` utility method for retrieving file inodes
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotUtils.java	Removes `getINode` method (moved to IOUtils)
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/SstFileInfo.java	Updates `getFilePath` to use getter method instead of direct field access
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/RdbUtil.java	Updates to use relocated `getINode` from IOUtils
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServlet.java	Updates to use relocated `getINode` from IOUtils
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOmSnapshotUtils.java	Updates import to use relocated `getINode`
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotManager.java	Updates import to use relocated `getINode`
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMRatisSnapshots.java	Updates import to use relocated `getINode`
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/TestFullDiffComputer.java	Comprehensive unit tests for FullDiffComputer covering various delta computation scenarios
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/TestFileLinkDeltaFileComputer.java	Unit tests for FileLinkDeltaFileComputer covering link creation, directory management, and error handling

Comments suppressed due to low confidence (1)

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotUtils.java:77

Access of element annotated with VisibleForTesting found in production code.

    Object inode = getINode(file);

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-18T18:45:37Z

...-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/DeltaFileComputer.java

+   * @return an {@code Optional} containing a collection of pairs, where each pair consists of a
+   *         {@code Path} representing the delta file and an associated {@code SstFileInfo}, or
+   *         an empty {@code Optional} if no changes are found


The return type documentation is incorrect. The method returns a Collection directly, not an Optional containing a collection. The JavaDoc should state:
@return a collection of pairs, where each pair consists of a Path representing the delta file and an associated SstFileInfo

Remove the mention of "an empty Optional if no changes are found" as the method throws IOException when delta files cannot be computed.

Suggested change

* @return an {@code Optional} containing a collection of pairs, where each pair consists of a

* {@code Path} representing the delta file and an associated {@code SstFileInfo}, or

* an empty {@code Optional} if no changes are found

* @return a collection of pairs, where each pair consists of a {@code Path} representing the delta file

* and an associated {@code SstFileInfo}

Copilot · 2025-11-18T18:45:38Z

.../src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/FileLinkDeltaFileComputer.java

+  private final OMMetadataManager activeMetadataManager;
+  private final Consumer<SubStatus> activityReporter;
+  private Path deltaDir;
+  private AtomicInteger linkFileCounter = new AtomicInteger(0);


The linkFileCounter field is not declared as final, which could lead to unexpected behavior if the reference is accidentally reassigned. Consider making it final:

private final AtomicInteger linkFileCounter = new AtomicInteger(0);

Suggested change

private AtomicInteger linkFileCounter = new AtomicInteger(0);

private final AtomicInteger linkFileCounter = new AtomicInteger(0);

Copilot · 2025-11-18T18:45:38Z

.../src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/FileLinkDeltaFileComputer.java

+    do {
+      link = deltaDir.resolve(linkFileCounter.incrementAndGet() + extension);
+      try {
+        Files.createLink(link, source);
+        createdLink = true;
+      }  catch (FileAlreadyExistsException ignored) {
+        LOG.info("File for source {} already exists: at {}. Will attempt to create link with a different path", source,
+            link);
+      }
+    } while (!createdLink);


The infinite loop with do-while (!createdLink) lacks a safety mechanism to prevent infinite loops in case of persistent failures. Consider adding a maximum retry limit to prevent resource exhaustion:

int maxRetries = 1000; int attempts = 0; do { if (++attempts > maxRetries) { throw new IOException("Failed to create link after " + maxRetries + " attempts for path: " + source); } link = deltaDir.resolve(linkFileCounter.incrementAndGet() + extension); try { Files.createLink(link, source); createdLink = true; } catch (FileAlreadyExistsException ignored) { LOG.info("File for source {} already exists: at {}. Will attempt to create link with a different path", source, link); } } while (!createdLink);

this shouldn't happen ideally. Not required

Copilot · 2025-11-18T18:45:39Z

.../src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/FileLinkDeltaFileComputer.java

+      try {
+        Files.createLink(link, source);
+        createdLink = true;
+      }  catch (FileAlreadyExistsException ignored) {


Extra space before catch keyword. Should be:

} catch (FileAlreadyExistsException ignored) {

Suggested change

} catch (FileAlreadyExistsException ignored) {

} catch (FileAlreadyExistsException ignored) {

Copilot · 2025-11-18T18:45:39Z

.../src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/FileLinkDeltaFileComputer.java

+        fromSnapshot.getBucketName());
+    return computeDeltaFiles(fromSnapshot, toSnapshot, tablesToLookup,
+        tablePrefixInfo).map(Map::values).orElseThrow(() -> new IOException(String.format(
+            "Failed to compute delta files for snapshots %s and %s tablesToLookup : %s", fromSnapshot, toSnapshot,


The error message contains inconsistent formatting with a missing space before "tablesToLookup". Should be:

"Failed to compute delta files for snapshots %s and %s. tablesToLookup: %s"

or

"Failed to compute delta files for snapshots %s and %s, tablesToLookup: %s"

Suggested change

"Failed to compute delta files for snapshots %s and %s tablesToLookup : %s", fromSnapshot, toSnapshot,

"Failed to compute delta files for snapshots %s and %s. tablesToLookup: %s", fromSnapshot, toSnapshot,

Copilot · 2025-11-18T18:45:42Z

...e-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/FullDiffComputer.java

+      Path fromSnapshotPath = fromSnapshot.getMetadataManager().getStore().getDbLocation().getAbsoluteFile().toPath();
+      Path toSnapshotPath = toSnapshot.getMetadataManager().getStore().getDbLocation().getAbsoluteFile().toPath();


Access of element annotated with VisibleForTesting found in production code.

Suggested change

Path fromSnapshotPath = fromSnapshot.getMetadataManager().getStore().getDbLocation().getAbsoluteFile().toPath();

Path toSnapshotPath = toSnapshot.getMetadataManager().getStore().getDbLocation().getAbsoluteFile().toPath();

Path fromSnapshotPath = fromSnapshot.getMetadataManager().getStore().getDbPath();

Path toSnapshotPath = toSnapshot.getMetadataManager().getStore().getDbPath();

removed the annotation not relevant anymore

Copilot · 2025-11-18T18:45:43Z

...e-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/FullDiffComputer.java

+        .getStore()).getDb().getManagedRocksDb(), tablesToLookUp), tablesToLookUp, tablePrefixInfo);
+  }
+
+  static Set<SstFileInfo> getSSTFileSetForSnapshot(OmSnapshot snapshot, Set<String> tablesToLookUp,
+      TablePrefixInfo tablePrefixInfo) {
+    return filterRelevantSstFiles(RdbUtil.getSSTFilesForComparison(((RDBStore)snapshot.getMetadataManager().getStore())
+            .getDb().getManagedRocksDb(), tablesToLookUp), tablesToLookUp, tablePrefixInfo);


Access of element annotated with VisibleForTesting found in production code.

Suggested change

.getStore()).getDb().getManagedRocksDb(), tablesToLookUp), tablesToLookUp, tablePrefixInfo);

}

static Set<SstFileInfo> getSSTFileSetForSnapshot(OmSnapshot snapshot, Set<String> tablesToLookUp,

TablePrefixInfo tablePrefixInfo) {

return filterRelevantSstFiles(RdbUtil.getSSTFilesForComparison(((RDBStore)snapshot.getMetadataManager().getStore())

.getDb().getManagedRocksDb(), tablesToLookUp), tablesToLookUp, tablePrefixInfo);

.getStore()).getDb(), tablesToLookUp), tablesToLookUp, tablePrefixInfo);

}

static Set<SstFileInfo> getSSTFileSetForSnapshot(OmSnapshot snapshot, Set<String> tablesToLookUp,

TablePrefixInfo tablePrefixInfo) {

return filterRelevantSstFiles(RdbUtil.getSSTFilesForComparison(((RDBStore)snapshot.getMetadataManager().getStore())

.getDb(), tablesToLookUp), tablesToLookUp, tablePrefixInfo);

removed the annotation not relevant anymore

Copilot · 2025-11-18T18:45:43Z

...e-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/FullDiffComputer.java

+
+  static Map<Object, SstFileInfo> getSSTFileMapForSnapshot(OmSnapshot snapshot,
+      Set<String> tablesToLookUp, TablePrefixInfo tablePrefixInfo) throws IOException {
+    return filterRelevantSstFiles(RdbUtil.getSSTFilesWithInodesForComparison(((RDBStore)snapshot.getMetadataManager()


Access of element annotated with VisibleForTesting found in production code.

Suggested change

return filterRelevantSstFiles(RdbUtil.getSSTFilesWithInodesForComparison(((RDBStore)snapshot.getMetadataManager()

return filterRelevantSstFiles(RdbUtil.getSSTFilesForComparison(((RDBStore)snapshot.getMetadataManager()

removed the annotation not relevant anymore

Copilot · 2025-11-18T18:45:43Z

...e-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/FullDiffComputer.java

+    return filterRelevantSstFiles(RdbUtil.getSSTFilesForComparison(((RDBStore)snapshot.getMetadataManager().getStore())
+            .getDb().getManagedRocksDb(), tablesToLookUp), tablesToLookUp, tablePrefixInfo);


Access of element annotated with VisibleForTesting found in production code.

Suggested change

return filterRelevantSstFiles(RdbUtil.getSSTFilesForComparison(((RDBStore)snapshot.getMetadataManager().getStore())

.getDb().getManagedRocksDb(), tablesToLookUp), tablesToLookUp, tablePrefixInfo);

// Use the production-safe method and convert its values to a Set

Map<Object, SstFileInfo> sstFileMap = RdbUtil.getSSTFilesWithInodesForComparison(

((RDBStore)snapshot.getMetadataManager().getStore()).getDb().getManagedRocksDb(), tablesToLookUp);

return filterRelevantSstFiles(sstFileMap, tablesToLookUp, tablePrefixInfo);

removed the annotation not relevant anymore

Copilot · 2025-11-18T18:45:44Z

...e-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/FullDiffComputer.java

+    return filterRelevantSstFiles(RdbUtil.getSSTFilesForComparison(((RDBStore)snapshot.getMetadataManager().getStore())
+            .getDb().getManagedRocksDb(), tablesToLookUp), tablesToLookUp, tablePrefixInfo);


Access of element annotated with VisibleForTesting found in production code.

Suggested change

return filterRelevantSstFiles(RdbUtil.getSSTFilesForComparison(((RDBStore)snapshot.getMetadataManager().getStore())

.getDb().getManagedRocksDb(), tablesToLookUp), tablesToLookUp, tablePrefixInfo);

// Use the public method getSSTFilesWithInodesForComparison instead of the @VisibleForTesting method.

return filterRelevantSstFiles(

RdbUtil.getSSTFilesWithInodesForComparison(

((RDBStore)snapshot.getMetadataManager().getStore()).getDb().getManagedRocksDb(),

tablesToLookUp),

tablesToLookUp, tablePrefixInfo);

removed the annotation not relevant anymore

jojochuang · 2025-11-18T18:41:26Z

.../src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/FileLinkDeltaFileComputer.java

+ * creating hard links for delta representation, and reporting activity
+ * during the computation process.
+ */
+public abstract class FileLinkDeltaFileComputer implements DeltaFileComputer {


apart from test code, where does it get used?

This would be used in the PR #9312

jojochuang · 2025-11-18T21:21:11Z

...e-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/FullDiffComputer.java

+  Optional<Map<Path, Pair<Path, SstFileInfo>>> computeDeltaFiles(SnapshotInfo fromSnapshotInfo,
+      SnapshotInfo toSnapshotInfo, Set<String> tablesToLookup, TablePrefixInfo tablePrefixInfo) throws IOException {
+    try (UncheckedAutoCloseableSupplier<OmSnapshot> fromSnapHandle = getSnapshot(fromSnapshotInfo);
+         UncheckedAutoCloseableSupplier<OmSnapshot> toSnapHandle = getSnapshot(toSnapshotInfo)) {


snapshot cache acquires read lock, so the order of lock shouldn't matter.

Change-Id: If84ba7867beb780747fee43ea4535309e737e0ab

Change-Id: Ib3f53fbd48444a6ac5da037bb163b9d83b69fca1

Change-Id: I4174c1f303baef2542d69dc5a18c512e8ae2251b

jojochuang

LGTM this PR adds a set of utility classes to encapsulate delta files for snapshot diff operations.

jojochuang · 2025-11-18T21:29:22Z

.../src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/FileLinkDeltaFileComputer.java

+  private final OmSnapshotManager omSnapshotManager;
+  private final OMMetadataManager activeMetadataManager;
+  private final Consumer<SubStatus> activityReporter;
+  private Path deltaDir;


deltaDir is a path to a temporary directory whose life cycle is managed by an instance of this class.

jojochuang · 2025-11-18T21:51:34Z

...e-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/diff/delta/FullDiffComputer.java

+
+/**
+ * FullDiffComputer is a specialized implementation of FileLinkDeltaFileComputer
+ * that computes the delta files between two snapshots. It identifies the differences


i think the phrase "delta file" needs a more clear definition.
That is,

files that appear in the "to" snapshot but not in "from"

files that appear in "from" but not in "to"

Will update this javadoc in the next PR since the CI already passed.

swamirishi added 30 commits October 8, 2025 09:39

HDDS-13765. SnapshotLocalData yaml should also track snapshotId

6e241f6

Change-Id: Iba47aeb21663dfa407ab71339cef02c0d74b49f2

HDDS-13627. In memory Manager for Snapshot Local Data

a869500

Change-Id: Ifd2feca1fddb144e4955db025f0b15a2ab1f3bfe

HDDS-13627. In memory Manager for Snapshot Local Data

252d338

Change-Id: I34536ff06efb7d5a4942853f0fd83942ab398b5f

HDDS-13767. Refactor SnapshotLocalDataYaml related code into OmSnapsh…

4099bc6

…otLocalDataManager Change-Id: I34536ff06efb7d5a4942853f0fd83942ab398b5f

HDDS-13767. Fix pmd

e02670c

Change-Id: I32bcaf2a1fb290f1790c02872a0230cd65586636

HDDS-13627. Fix checkstyle

79580e9

Change-Id: I985170e38fb8beeb784048e85a08a4c79e1aec97

Merge remote-tracking branch 'apache/master' into HEAD

2a331ef

Change-Id: I33e6e6e825bf23c323ad7ed593d800a11720fa4f

Merge remote-tracking branch 'apache/master' into HEAD

c4f69e2

Change-Id: If30b2c766db82adde72145c8ecd3e590ef54cc2d

HDDS-13627. Add tests

afbc592

Change-Id: Id3f2c49050bc3476b9e0f5f51dacb6d9acc4c2f7

HDDS-13783. Implement locks for OmSnapshotLocalDataManager

70ac2c7

Change-Id: I432960725b4c6c55aa906b5780cc3027e41e10db

HDDS-13783. Implement locks for OmSnapshotLocalDataManager

b554cc7

Change-Id: I3c5514e5bbd251a2b5297d8f074cfde5c71fa543

HDDS-13783. Refactor inline variable

49eccfa

Change-Id: Ib5a9e6c91bdccba17820263c47eaf2c8400e930d

HDDS-13627. Refactor map data structure

51eda04

Change-Id: Ica36e0615c7bc6aa9b6a7f6fafafd0f830d4bafb

Merge remote-tracking branch 'origin/HDDS-13627' into HEAD

25f766c

Change-Id: I26b66f266bb7677e4b1078f5fcd9f2ce3a651a70 # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OmSnapshotLocalDataManager.java

HDDS-13783. Add more condition to upsert

96689fa

Change-Id: I1d93dbc048a42cc55ff1f8ffa420e52f967527b8

HDDS-13783. Add java doc comment

0674299

Change-Id: I34202928a7a367dd0a1e57219317ff34de352b78

HDDS-13783. Add java doc comment

5d9fc49

Change-Id: Iad6f26cb71ec921c51ee2d138745df1a2663533f

HDDS-13783. Implement full lock

2d88176

Change-Id: Ic5f7e249cfb9cb3973cbcd4abd36b22a6ff8f5aa

HDDS-13783. Refactor and move modify method into WritableOmSnapshotLo…

a3c4c69

…calDataProvider Change-Id: I3a004b4b435075a4348960aeed642e8da71e7e72

HDDS-13783. Make full lock non static

686d0c7

Change-Id: I06990bc9ab8fc7e1eb7bec255646a650bd8c35fe

HDDS-13783. Fix remove

491a54b

Change-Id: I4c6c61c83aa9fadab8ecef854b99dcc0a89a2208

HDDS-13627. Fix findbugs

5e69ee9

Change-Id: I0e476322372a302572f1fe79cbf2e874bfeac2ed

HDDS-13785. Remove orphan versions from SnapshotLocalData Yaml file

d36622a

Change-Id: I31004e0c95dad64411c6fe848501a82f2f773cba

HDDS-13785. Fix findbugs

ee213d1

Change-Id: Id317c8b56e8b25c122b68eaf96599b9690d08f79

HDDS-13627. Fix tests

a95604e

Change-Id: Ie5e5f3dab4324103e8855dd15619d7755f0422e6

HDDS-13627. remove checksum interface

5a90fcf

Change-Id: I55bd5c3ef7fc32910a9111328638de2edffcd541

HDDS-13627. Fix test failures

20d7d6a

Change-Id: I880997d3eebdf378f14c203c61c2d63b2d17552e

HDDS-13785. Set defrag flag on previous snapshotId update

ae655cb

Change-Id: I13ba8e2fd012a3c964d657e83496c93a4f55a3be

swamirishi added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label Nov 11, 2025

swamirishi requested review from jojochuang, sadanand48 and smengcl November 11, 2025 23:44

swamirishi added 11 commits November 12, 2025 09:48

HDDS-13912. Fix create link logic

8523dea

Change-Id: Iaad718f0c7bc23c67650144c0c7ac26f0060a437

HDDS-13912. Fix findbugs

9ee0298

Change-Id: Id2be646eeb0aa1d264211bf2950803bcb2839ce9

Merge remote-tracking branch 'origin/HDDS-13867' into HEAD

121977e

Change-Id: Ie3d199a6401b01af35e63781e2b6656d6aa1d5e9

HDDS-13867. Fix bug

9736247

Change-Id: I1d7aded575ac6daaa5f78b3be9f1821d94cb24c6

HDDS-13867. Add test

2167b03

Change-Id: Iba0dd30952d17d11eae900213437aa4c28564448

HDDS-13867. Address review comments

08156e4

Change-Id: Idba640c859ed4906842f9fafebf04825409053ef

Merge remote-tracking branch 'origin/HDDS-13867' into HEAD

84a0352

Change-Id: I47fca4c2ca1738b533992851fa54291cb109cc39

HDDS-13912. Address review comments

8890252

Change-Id: Id07b78b650ce58fcc71bbac478d085b139123b02

HDDS-13912. Fix interface

1067935

Change-Id: Iacf5dd1fd2013d6203ea7003645cb79433186519

HDDS-13912. Fix exception message

b086376

Change-Id: I22a212c84a3dbcd66a8e61a940d4df239b73133f

swamirishi mentioned this pull request Nov 17, 2025

HDDS-13929. Modularise Snapshot Delta file computer (Efficient Diff) #9312

Merged

swamirishi added 2 commits November 17, 2025 23:41

Merge remote-tracking branch 'apache/master' into HEAD

ed8ee79

Change-Id: Ib313080b2ffa2f6e7545ae817f5e456cabc5dba7

jojochuang requested a review from Copilot November 18, 2025 18:34

Copilot started reviewing on behalf of jojochuang November 18, 2025 18:35 View session

Copilot finished reviewing on behalf of jojochuang November 18, 2025 18:38

Copilot AI reviewed Nov 18, 2025

View reviewed changes

jojochuang reviewed Nov 18, 2025

View reviewed changes

swamirishi added 3 commits November 18, 2025 16:38

HDDS-13912. Remove visible for testing annotation

7c27532

Change-Id: If84ba7867beb780747fee43ea4535309e737e0ab

HDDS-13912. Fix javadoc

025411d

Change-Id: Ib3f53fbd48444a6ac5da037bb163b9d83b69fca1

HDDS-13912. Address review coments

34fa1b4

Change-Id: I4174c1f303baef2542d69dc5a18c512e8ae2251b

jojochuang approved these changes Nov 18, 2025

View reviewed changes

jojochuang marked this pull request as ready for review November 18, 2025 22:02

jojochuang merged commit 00dd115 into apache:master Nov 19, 2025
55 checks passed

	private AtomicInteger linkFileCounter = new AtomicInteger(0);
	private final AtomicInteger linkFileCounter = new AtomicInteger(0);

	} catch (FileAlreadyExistsException ignored) {
	} catch (FileAlreadyExistsException ignored) {

	"Failed to compute delta files for snapshots %s and %s tablesToLookup : %s", fromSnapshot, toSnapshot,
	"Failed to compute delta files for snapshots %s and %s. tablesToLookup: %s", fromSnapshot, toSnapshot,

		Path fromSnapshotPath = fromSnapshot.getMetadataManager().getStore().getDbLocation().getAbsoluteFile().toPath();
		Path toSnapshotPath = toSnapshot.getMetadataManager().getStore().getDbLocation().getAbsoluteFile().toPath();

	return filterRelevantSstFiles(RdbUtil.getSSTFilesWithInodesForComparison(((RDBStore)snapshot.getMetadataManager()
	return filterRelevantSstFiles(RdbUtil.getSSTFilesForComparison(((RDBStore)snapshot.getMetadataManager()

		return filterRelevantSstFiles(RdbUtil.getSSTFilesForComparison(((RDBStore)snapshot.getMetadataManager().getStore())
		.getDb().getManagedRocksDb(), tablesToLookUp), tablesToLookUp, tablePrefixInfo);

-    return filterRelevantSstFiles(RdbUtil.getSSTFilesForComparison(((RDBStore)snapshot.getMetadataManager().getStore())
-            .getDb().getManagedRocksDb(), tablesToLookUp), tablesToLookUp, tablePrefixInfo);
+    // Use the production-safe method and convert its values to a Set
+    Map<Object, SstFileInfo> sstFileMap = RdbUtil.getSSTFilesWithInodesForComparison(
+        ((RDBStore)snapshot.getMetadataManager().getStore()).getDb().getManagedRocksDb(), tablesToLookUp);
+    return filterRelevantSstFiles(sstFileMap, tablesToLookUp, tablePrefixInfo);

HDDS-13912. Modularise Snapshot Delta file computer (Full Diff) #9283

HDDS-13912. Modularise Snapshot Delta file computer (Full Diff) #9283

Uh oh!

Conversation

swamirishi commented Nov 11, 2025

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects