HDDS-6577. Configurations to reserve HDDS volume space. #3484
Conversation
umamaheswararao
left a comment
umamaheswararao
left a comment
There was a problem hiding this comment.
@aswinshakil Thanks for working on this patch. I have dropped few minors. Please check. Thanks
| // bytes in in-memory container status. | ||
| // bytes in in-memory container status and used space in volume. | ||
| containerData.decrPendingDeletionBlocks(deletedBlocksCount); | ||
| containerData.decrBlockCount(deletedBlocksCount); |
There was a problem hiding this comment.
Isn't it a good idea to decrement the space in containerData.decrBytesUsed ?
So, that increment and decrement will be in same place?
There was a problem hiding this comment.
I thought it would be better to decrement Container Data and Volume Usage separately. As the former is just intended to decrement the container data used bytes.
There was a problem hiding this comment.
But for increment we did inside containerData even for volume. I am wondering if we do all inside container data may be consistent? ( Unless we have any other potential issues in doing that.)
There was a problem hiding this comment.
For increment we had incrWriteBytes() which is used in updateWriteStats() and is different from incrBytesUsed(), both are done as two operations inside updateWriteStats(). We can do something like create a function decrWriteBytes() and then call decrementUsedSpace(), but it will be same instead of calling containerData.getVolume().decrementUsedSpace() we will be calling containerData.decrWriteBytes()
There was a problem hiding this comment.
ok thanks for the explanation.
...ainer-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeUsage.java
Show resolved
Hide resolved
* master: (34 commits) HDDS-6868 Add S3Auth information to thread local (apache#3527) HDDS-6877. Keep replication port unchanged when restarting datanode in MiniOzoneCluster (apache#3510) HDDS-6907. OFS should create buckets with FILE_SYSTEM_OPTIMIZED layout. (apache#3528) HDDS-6875. Migrate parameterized tests in hdds-common to JUnit5 (apache#3513) HDDS-6924. OBJECT_STORE isn't flat namespaced (apache#3533) HDDS-6899. [EC] Remove warnings and errors from console during online reconstruction of data. (apache#3522) HDDS-6695. Enable SCM Ratis by default for new clusters only (apache#3499) HDDS-4123. Integrate OM Open Key Cleanup Service Into Existing Code (apache#3319) HDDS-6882. Correct exit code for invalid arguments passed to command-line tools. (apache#3517) HDDS-6890. EC: Fix potential wrong replica read with over-replicated container. (apache#3523) HDDS-6902. Duplicate mockito-core entries in pom.xml (apache#3525) HDDS-6752. Migrate tests with rules in hdds-server-scm to JUnit5 (apache#3442) HDDS-6806. EC: Implement the EC Reconstruction coordinator. (apache#3504) HDDS-6829. Limit the no of inflight replication tasks in SCM. (apache#3482) HDDS-6898. [SCM HA finalization] Modify acceptance test configuration to speed up test finalization (apache#3521) HDDS-6577. Configurations to reserve HDDS volume space. (apache#3484) HDDS-6870 Clean up isTenantAdmin to use UGI (apache#3503) HDDS-6872. TestAuthorizationV4QueryParser should pass offline (apache#3506) HDDS-6840. Add MetaData volume information to the SCM and OM - UI (apache#3488) HDDS-6697. EC: ReplicationManager - create class to detect EC container health issues (apache#3512) ...
…pache#3484 Change-Id: I47ed0bc40b828e499a71b6ec7fa753e2bb087509
2 participants
What changes were proposed in this pull request?
MaxTotalWalSizefor DN containers to2MB. RocksDB by default pre-allocates141MBWAL file for every container and it only uses less than2MBof it.CachingSpaceUsageSourceperiodically refreshed the used space of a Volume. (hdds.datanode.du.refresh.period=1hforDUandhdds.datanode.df.refresh.period=5mforDedicatedDiskSpaceUsage). During the interval used space is not updated, Hence the volume usage goes beyond the reserved space(hdds.datanode.dir.du.reserved)What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6577
How was this patch tested?
This patch was tested manually using the docker cluster.