HDDS-9426. Calculate Exclusive size for deep cleaned snapshot's deleted directories

[aswinshakil]

What changes were proposed in this pull request?

The original patch #5579 was reverted due to incompatible proto.lock changes and hence reverted in #6051. The proto.lock is now merged to master in #6080 . Hence re-raising the PR. Copying the original PR description here for context.

Original PR description

In #5339, We expanded the snapshot's deletedDirTable into fileTable to calculate the exclusive size for the files under the deleted directories. The problem with that approach is that if we alter the fileTable and dirTable of a snapshot, it would change the snapDiff result and yield undesirable results. For eg, Consider a deleted directory structure as below,

dir1 --- subdir1 --- file1 to file100
    | 
     ---- subdir2 --- file1 to file100

Instead of snapDiff saying DELETED dir1, It would say
DELETED subdir1
DELETED subdir2
DELETED subdir1/file1 to file100
DELETED subdir2/file1 to file100

In the new approach in this patch, We do an in-memory walk for each deleted directory and deep clean them, and also calculate the exclusive size for files when doing so. For each file, we do the same as #5301

The reason we are doing a deep clean, There could be files under a directory trapped in between snapshots. Consider this case,

Create dir1
Create snapshot1
Create 1000 files under dir1 --> These files are trapped and not deleted unless the snapshot itself is deleted.
Delete dir1
Create snapshot2

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-9426

How was this patch tested?

Added Integration test.

[hemantk-12]

Thanks @aswinshakil for bringing it back.

LGTM.

[adoroszlai]

Thanks @aswinshakil for re-working the proto definition to be backwards compatible.

Thanks @hemantk-12 for the review.