HDDS-12887. Added Deletion Progress metrics to OM Web UI. #8655

SaketaChalamchala · 2025-06-18T18:09:43Z

What changes were proposed in this pull request?

To provide visibility into the Key Deleting Service progress, added a "Deletion Progress" section to the OM Web UI displaying the cumulative deletion progress (no. of keys and size) in the last 1 day as well as the KDS service status, Last run details and the next run information.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-12887

How was this patch tested?

Unit Tests for metrics

jojochuang · 2025-06-24T05:43:56Z

Unable to open the screenshot.

sadanand48 · 2025-06-24T07:26:19Z

hadoop-ozone/ozone-manager/src/main/resources/webapps/ozoneManager/om-overview.html

+                          <th>Reclaimed Size</th>
+                          <th>#Reclaimed Keys</th>
+                          <th>#Iterated Keys</th>
+                          <th>#NotReclaimable Keys</th>


NotReclaimable keys makes sense in context of snapshots. We could change it to

Suggested change

<th>#NotReclaimable Keys</th>

<th>#NotReclaimable Keys (Referred by snapshots)</th>

SaketaChalamchala · 2025-07-02T22:27:40Z

This PR might need to be merged after #8677

jojochuang · 2025-07-24T15:18:11Z

Will merge #8677 today to unblock this one.

…he last 24h.

…mtrics and KDS service status.

…eb UI.

…ache#8677.

Copilot

Pull Request Overview

This PR adds a "Deletion Progress" section to the OM Web UI to provide visibility into the Key Deleting Service progress, displaying cumulative deletion metrics for the last 24 hours along with service status and run information.

Key changes include:

Added comprehensive deletion metrics tracking for both Active Object Store (AOS) and snapshot operations
Enhanced the OM Web UI with a new deletion progress section showing metrics and service status
Extended the Key Deleting Service to track and report detailed deletion statistics

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
TestKeyDeletingService.java	Added unit tests for deletion metrics and updated existing test methods
ozoneManager.js	Added utility functions for data formatting and JMX query for DeletingServiceMetrics
om-overview.html	Added new UI section displaying deletion progress metrics and service status
KeyDeletingService.java	Enhanced to track deletion statistics and update metrics after task completion
PendingKeysDeletion.java	Extended to include key block replicated size and not reclaimable key count
KeyManagerImpl.java	Updated to populate additional metrics data for pending deletions
DeletingServiceMetrics.java	Added new metrics for 24-hour intervals, service state, and last run statistics
BackgroundService.java	Added task completion callback to support metrics updates

Comments suppressed due to low confidence (1)

hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/service/TestKeyDeletingService.java:825

The test is passing null for the keyBlockReplicatedSize parameter which means the deletion size metrics won't be properly tested. Consider providing a mock map to verify size tracking functionality.

        keyDeletingService.processKeyDeletes(blockGroups, keysToModify, renameEntriesToBeDeleted, null, null, null);

Copilot · 2025-07-30T19:07:42Z

hadoop-ozone/ozone-manager/src/main/resources/webapps/ozoneManager/ozoneManager.js

+            // Add JMX query to fetch DeletingServiceMetrics data
+            $http.get("jmx?qry=Hadoop:service=OzoneManager,name=DeletingServiceMetrics")
+                .then(function (result) {
+                    if (result.data.beans && result.data.beans.length > 0) {


Missing closing brace for the function that contains this if statement. The function appears to be incomplete and will cause a syntax error.

...ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/KeyDeletingService.java

jojochuang · 2025-07-31T06:51:41Z

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/DeletingServiceMetrics.java

+    this.kdsCurRunTimestamp.set(timestamp);
+  }
+
+  private void resetMetrics() {


why reset metrics after 24hours? usually metrics are reset when restart, and that's good enough.

We already collect deletion metrics since OM uptime. The thought was on OM UI it showing the progress of deletion over the last 24h would be more interesting for someone who has performed a delete and wants to track how the delete is proceeding.

jojochuang

Had a quick look and I think this is ready to go.

jojochuang · 2025-07-31T07:06:59Z

...ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/KeyDeletingService.java

+              aosDeletionStats.updateDeletionStats(purgeResult.getKey().getKey(), purgeResult.getKey().getValue(),
+                  keyBlocksList.size() + pendingKeysDeletion.getNotReclaimableKeyCount(),
+                  pendingKeysDeletion.getNotReclaimableKeyCount());
+            } else {
+              snapshotDeletionStats.updateDeletionStats(purgeResult.getKey().getKey(), purgeResult.getKey().getValue(),
+                  keyBlocksList.size() + pendingKeysDeletion.getNotReclaimableKeyCount(),
+                  pendingKeysDeletion.getNotReclaimableKeyCount());


it seems these two cases are almost exactly the same except the object itself is aosDeletionStats or snapshotDeletionStats. A refactoring is recommended though not required.

hadoop-ozone/ozone-manager/src/main/resources/webapps/ozoneManager/ozoneManager.js

jojochuang

loos good to me.
This PR should ideally split into two parts due to its complexity: metric implementation, and frontend code.

Will merge later.

…ager/ozoneManager.js

jojochuang · 2025-08-01T00:34:33Z

Thanks @SaketaChalamchala !

…apache#8655) Co-authored-by: Wei-Chiu Chuang <jojochuang@gmail.com> (cherry picked from commit 37589a4) Conflicts: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/DeletingServiceMetrics.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/KeyDeletingService.java hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/service/TestKeyDeletingService.java Change-Id: Ic874a0bb516712100b5bc5a584a2a4a54858d1da

jojochuang added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label Jun 24, 2025

sadanand48 reviewed Jun 24, 2025

View reviewed changes

jojochuang requested a review from sadanand48 July 7, 2025 15:31

SaketaChalamchala force-pushed the HDDS-12887 branch from 648bc9e to 9ef29dd Compare July 26, 2025 06:43

jojochuang mentioned this pull request Jul 30, 2025

HDDS-13320. BackgroundService should wait for the completion of run before triggering next run #8677

Merged

saketa added 8 commits July 30, 2025 11:18

HDDS-12887. Added service status metrics and cumulative metrics for t…

6ff8341

…he last 24h.

HDDS-12887. Added deleted key replicated size to the metrics.

c220eb1

HDDS-12887. Added Deletion Progress section to OM UI with cumulative …

e06f132

…mtrics and KDS service status.

HDDS-12887. Added last run details metrics and published them to OM W…

c5fb9ee

…eb UI.

HDDS-12887. Updated UI format. Fixed last run metric collection.

f59fe42

HDDS-12887. Fixed checkstyle and findbugs errors.

1537edf

HDDS-12887. Added tests for metrics.

1f08460

HDDS-12887. Updated UI and patch to highlight 24h summary based on ap…

3f26660

…ache#8677.

SaketaChalamchala force-pushed the HDDS-12887 branch from 9ef29dd to 3f26660 Compare July 30, 2025 19:06

SaketaChalamchala marked this pull request as ready for review July 30, 2025 19:07

Copilot AI review requested due to automatic review settings July 30, 2025 19:07

Copilot AI reviewed Jul 30, 2025

View reviewed changes

jojochuang reviewed Jul 31, 2025

View reviewed changes

HDDS-12887. Simplified updating AOS and snapshot stats.

179f04e

jojochuang reviewed Jul 31, 2025

View reviewed changes

hadoop-ozone/ozone-manager/src/main/resources/webapps/ozoneManager/ozoneManager.js Outdated Show resolved Hide resolved

jojochuang approved these changes Jul 31, 2025

View reviewed changes

Update hadoop-ozone/ozone-manager/src/main/resources/webapps/ozoneMan…

8d5e137

…ager/ozoneManager.js

jojochuang merged commit 37589a4 into apache:master Aug 1, 2025
1 check passed

	<th>#NotReclaimable Keys</th>
	<th>#NotReclaimable Keys (Referred by snapshots)</th>

HDDS-12887. Added Deletion Progress metrics to OM Web UI. #8655

HDDS-12887. Added Deletion Progress metrics to OM Web UI. #8655

Uh oh!

Conversation

SaketaChalamchala commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

jojochuang commented Jun 24, 2025

Uh oh!

sadanand48 Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

SaketaChalamchala commented Jul 2, 2025

Uh oh!

jojochuang commented Jul 24, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jojochuang Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SaketaChalamchala Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

jojochuang Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jojochuang commented Aug 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

SaketaChalamchala commented Jun 18, 2025 •

edited

Loading

jojochuang Jul 31, 2025 •

edited

Loading