HDDS-12437. [DiskBalancer] Estimate the total size pending to move before disk usage becomes even

[Gargi-jais11]

What changes were proposed in this pull request?

It will be an estimation value, due to there are other activities going on, such as block deletion, new container creation, container replica deletion, new data ingestion.

This value will be an indicator for roughly how much time are still needed for the disk usage to become even.

Should Include this value in status report too.

The estimated time pending before disk usage become even is given for the happy path in status report.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-12437

How was this patch tested?

Unit Test is written for the function in testDiskBalancerService#testCalculateBytesToMove.

Tested it manually by running locally on docker-cluster.

bash-5.1$ ozone admin datanode diskbalancer status
Status result:
Datanode                            VolumeDensity             Status          Threshold(%)    BandwidthInMB   Threads      SuccessMove  FailureMove  EstBytesToMove(MB) EstTimeLeft(min)
ozone-datanode-2.ozone_default      0.001334742544789370      RUNNING         0.0002          10              5            5            0            816                2           
ozone-datanode-1.ozone_default      0.001486062001891789      RUNNING         0.0002          10              5            5            0            3185               6           

Note: Estimated time left is calculated based on the estimated bytes to move and the configured disk bandwidth.

bash-5.1$ ozone admin datanode diskbalancer status
Status result:
Datanode                            VolumeDensity             Status          Threshold(%)    BandwidthInMB   Threads      SuccessMove  FailureMove  EstBytesToMove(MB) EstTimeLeft(min)
ozone-datanode-2.ozone_default      0.001334742544789370      RUNNING         0.0002          10              5            8            2            683                2           
ozone-datanode-1.ozone_default      0.001486062001891789      RUNNING         0.0002          10              5            5            3            763                2           

Note: Estimated time left is calculated based on the estimated bytes to move and the configured disk bandwidth.

[ChenSammi]

@Gargi-jais11 , let's show EstBytesToMove(MB) together with the estimation time. And add a note at the command final, explain that estimation time is calculated based on estimation bytes to move and bandwidth.

[Gargi-jais11]

@Gargi-jais11 , let's show EstBytesToMove(MB) together with the estimation time. And add a note at the command final, explain that estimation time is calculated based on estimation bytes to move and bandwidth.

Sure will do the above mentioned changes

[ChenSammi]

Why hard code the queue size to 2?

[Gargi-jais11]

Because if I don't hard code the queue size than when it checks the actual value of bytesToMove through calculateBytesToMove then everytime the queue size is 0 as for checking the queue size getTask is not called and when I was trying to call it then containerChoosingPolicy was throwing lots of errors to just check whether the calculateBytesToMove I did it hardcode

[ChenSammi]

You can pass the volumeSet and OzoneConfiguration to calculateBytesToMove() and make it public, so that you can directly call calculateBytesToMove from the unit test, and don't have to deal with the containerChoosingPolicy.

[Gargi-jais11]

I tried passing the volumeSet and OzoneConfiguration to calculateBytesToMove() without hard coding the queue size to 2 or as volume count but the bytesToMove is still returning to be 0 due to queuesize = 0 else after refactoring the unit test as you suggested on hard coding the queue size to 2 all test case are passing

[ChenSammi]

Please also update the console output.

[Gargi-jais11]

Please also update the console output.

done

[ChenSammi]

Thanks @Gargi-jais11 .