HDDS-12431. [DiskBalancer] Use committedBytes to reserve the space pre-allocated for container #8297
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Gargi-jais11
force-pushed
the
HDDS-12431
branch
from
1e6a23f to
8713702
Compare
ChenSammi
reviewed
Apr 22, 2025
...ervice/src/main/java/org/apache/hadoop/ozone/container/diskbalancer/DiskBalancerService.java
Outdated
Show resolved
Hide resolved
# Conflicts: # hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/diskbalancer/policy/DefaultVolumeChoosingPolicy.java
Gargi-jais11
force-pushed
the
HDDS-12431
branch
from
8713702 to
2ef1a6b
Compare
Gargi-jais11
force-pushed
the
HDDS-12431
branch
from
1baf681 to
88ab23c
Compare
ChenSammi
reviewed
Apr 23, 2025
...ervice/src/main/java/org/apache/hadoop/ozone/container/diskbalancer/DiskBalancerService.java
Outdated
Show resolved
Hide resolved
Gargi-jais11
force-pushed
the
HDDS-12431
branch
from
2bef754 to
d461c12
Compare
chungen0126
reviewed
Apr 23, 2025
Contributor
chungen0126
left a comment
chungen0126
left a comment
There was a problem hiding this comment.
Thanks @Gargi-jais11 for the patch.
| inProgressContainers.add(toBalanceContainer.getContainerID()); | ||
| deltaSizes.put(sourceVolume, deltaSizes.getOrDefault(sourceVolume, 0L) | ||
| - toBalanceContainer.getBytesUsed()); | ||
| deltaSizes.put(destVolume, deltaSizes.getOrDefault(destVolume, 0L) |
Contributor
There was a problem hiding this comment.
Since we are still using deltaSizes to choose volume, why not just keep this. But also increase committed bytes to avoid over allocated.
Contributor
Author
There was a problem hiding this comment.
We're using deltaSizes to reflect space expected to be freed on the source volume (though not yet deleted), and committedBytes to reflect space already reserved on the destination volume (e.g., due to pending container moves). This ensures chooseVolume() to consider both volume committedBytes and deltaSize , one for destVolume and one for srcVolume.
| inProgressContainers.remove(containerData.getContainerID()); | ||
| deltaSizes.put(sourceVolume, deltaSizes.get(sourceVolume) + | ||
| containerData.getBytesUsed()); | ||
| deltaSizes.put(destVolume, deltaSizes.get(destVolume) |
ChenSammi
reviewed
Apr 24, 2025
...ervice/src/main/java/org/apache/hadoop/ozone/container/diskbalancer/DiskBalancerService.java
Outdated
Show resolved
Hide resolved
ChenSammi
reviewed
Apr 24, 2025
...ervice/src/main/java/org/apache/hadoop/ozone/container/diskbalancer/DiskBalancerService.java
Outdated
Show resolved
Hide resolved
Gargi-jais11
force-pushed
the
HDDS-12431
branch
from
db997f6 to
f483261
Compare
ChenSammi
approved these changes
Apr 25, 2025
Contributor
ChenSammi
left a comment
ChenSammi
left a comment
There was a problem hiding this comment.
Thanks @Gargi-jais11 , and @chungen0126 for the review.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Currently it uses deltaSizes to record the pre-allocated and pre-deleted bytes of each volume due to there is selected container to move.
But deltaSizes is not aware by other activities, such as container creation, container moving command by container balancer. So there is chance that data volume will be over allocated due to this.
This task aims to use HddsVolume#committedBytes to replace deltaSizes, so that the space pre-allocated by disk balancer will be fully awared by HddsVolume.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-12431
How was this patch tested?
Passed existing tests.