HDDS-9159. [Snapshot] Implement snapshot disk usage: Referenced size

[smengcl]

What changes were proposed in this pull request?

A snapshot's referenced size after replication is identical to the bucket's point-in-time omBucketInfo.getUsedBytes().

Note Ozone currently only have the bucket size AFTER replication stored for quota counting. The referenced size before replication is an estimate, in order to keep create snapshot operation O(1).

• Snapshot referenced size is stored directly into SnapshotInfoTable during snapshot creation so as to avoid opening every single snapshot under a bucket just to display the size. (Later we need an admin command or upgrade mechanism in OM to populate the referencedSize field for older clusters.)
• Referenced size can now be shown under fs -ls (it was always 0 before this change):

bash-4.2$ ozone fs -mkdir -p ofs://om/vol1/buck2/dir3/subdir4/
bash-4.2$ ozone fs -put README.md ofs://om/vol1/buck2/dir3/subdir4/
bash-4.2$ ozone fs -createSnapshot ofs://om/vol1/buck2/ snap1
Created snapshot ofs://om/vol1/buck2/.snapshot/snap1
bash-4.2$ ozone fs -ls ofs://om/vol1/buck2/.snapshot/
Found 1 items
drwxrwxrwx   - hadoop hadoop       4068 2023-08-11 07:30 ofs://om/vol1/buck2/.snapshot/snap1
bash-4.2$

• referencedSize also shown with ozone sh snapshot ls /vol1/buck2:

bash-4.2$ ozone sh snapshot ls /vol1/buck2
[ {
  "volumeName" : "vol1",
  "bucketName" : "buck2",
  "name" : "snap1",
  "creationTime" : 1691778428556,
  "snapshotStatus" : "SNAPSHOT_ACTIVE",
  "snapshotId" : "6a3c9f54-1e7b-4516-bf8d-44cbe3309312",
  "snapshotPath" : "vol1/buck2",
  "checkpointDir" : "-6a3c9f54-1e7b-4516-bf8d-44cbe3309312",
  "referencedSize" : 4068
} ]

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-9159

How was this patch tested?

• Tested manually in Docker compose cluster.
• Tested via testFsLsSnapshot.

[umamaheswararao]

@swamirishi @hemantk-12 one of you take a look at it?

[hemantk-12]

Thanks for working on this @smengcl.

Overall looks good to me. Left some minor comments.

[hemantk-12]

LGTM

[swamirishi]

Thanks @smengcl for the patch.
nit: An assert statement should also be created in TestOMSnapshotCreateRequest#testValidateAndUpdateCache & TestOmSnapshotCreateResponse#testAddToDBBatch. Otherwise the patch looks good to me

[smengcl]

Thanks @smengcl for the patch. nit: An assert statement should also be created in TestOMSnapshotCreateRequest#testValidateAndUpdateCache & TestOmSnapshotCreateResponse#testAddToDBBatch. Otherwise the patch looks good to me

Added in TestOMSnapshotCreateRequest.

It is not making much sense to add the check in TestOmSnapshotCreateResponse since it does not concern the Request change in this PR. testAddToDBBatch is crafting its own OMSnapshotCreateResponse object just to test addToDBBatch and does not go through validateAndUpdateCache at all.

[hemantk-12]

LGTM.

[aswinshakil]

Thanks for the patch @smengcl, The changes look good to me. I have only a few minor comments. Otherwise LGTM! 👍

[aswinshakil]

LGTM. Pending CI

[smengcl]

Thanks @hemantk-12 @swamirishi @aswinshakil for reviewing this.