HDDS-11699. Remove unnecessary information about parts when downloading multipart files.

[juncevich]

What changes were proposed in this pull request?

HDDS-11699. Remove unnecessary information about parts when downloading multipart files

Whenever information was requested about a particular part of a multipart file, it was obtained about all parts and filtered on the Ozone Manager side. To reduce work with unnecessary information, the ability to ask for information about the current part was added.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11699

How was this patch tested?

It was tested manually and by perf tests. Perf tests shown performance improvements:
Reduce max CPU consumption approximately 3 times.
Reduce HEAP size approximately 3 times.
Average GC pauses approximately 2 times.

[adoroszlai]

Thanks @juncevich for the patch. Please take a look at failed tests:

https://github.com/juncevich/ozone/actions/runs/12272551726/job/34242975199#step:7:13

https://github.com/juncevich/ozone/actions/runs/12272551726/job/34242757028#step:7:18

[juncevich]

Thanks @juncevich for the patch. Please take a look at failed tests:

https://github.com/juncevich/ozone/actions/runs/12272551726/job/34242975199#step:7:13

https://github.com/juncevich/ozone/actions/runs/12272551726/job/34242757028#step:7:18

@adoroszlai Thank you. Will fix.

[ivandika3]

@juncevich Thanks for the patch. Left some comments.

High level comments

• Please add some tests to test the S3 get with part number
  • Unit tests
    • TestKeyManagerImpl to test the changes in KeyManagerImpl
  • Integration tests
    • TestOzoneRpcClientAbstract to test the getS3KeyDetails andgetS3PartKeyInfo
    • AbstractS3SDKV1Tests to test get object, specifying the part number.
      • GetObjectRequest can be specify the part number
  • Acceptance test
    • objectputget.robot: Add the get-object for with --part-number specified (see https://docs.aws.amazon.com/cli/latest/reference/s3api/get-object.html)
• Might need to add a new OzoneManagerVersion for compatibility.
  • See how RpcClient#listStatusLight implements it
• Please simplify the logic, the change should simply move the filtering logic from client (S3G) to server (OM) side

[juncevich]

@ivandika3 would you re-review this pr

[ivandika3]

@juncevich Thanks for the update.

Just looking briefly, seems #7558 (comment) has not been addressed?

I'll review the rest ASAP. Thanks again for iterating on this.

[juncevich]

@juncevich Thanks for the update.

Just looking briefly, seems #7558 (comment) has not been addressed?

I'll review the rest ASAP. Thanks again for iterating on this.

Hmm, yep, you're right. I definitely fixed it. I will find a place where I lost it.

[juncevich]

@juncevich Thanks for the update.

Just looking briefly, seems #7558 (comment) has not been addressed?

I'll review the rest ASAP. Thanks again for iterating on this.

Fixed. Maybe I have to recheck tests.

[ivandika3]

Thanks for the update. Overall LGTM. I left some minor comments.

[ivandika3]

Thanks for the update. LGTM +1.

Let's wait for a clean CI.

[adoroszlai]

Let's wait for a clean CI.

Please check failures:

Error:  Tests run: 29, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 197.004 s <<< FAILURE! - in org.apache.hadoop.ozone.om.TestKeyManagerImpl
Error:  org.apache.hadoop.ozone.om.TestKeyManagerImpl.testGetNotExistedPart  Time elapsed: 0.045 s  <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <0> but was: <5>
...
	at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:531)
	at org.apache.hadoop.ozone.om.TestKeyManagerImpl.testGetNotExistedPart(TestKeyManagerImpl.java:1591)

Error:  org.apache.hadoop.ozone.om.TestKeyManagerImpl.testGetParticularPart  Time elapsed: 0.039 s  <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <1> but was: <5>
...
	at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:531)
	at org.apache.hadoop.ozone.om.TestKeyManagerImpl.testGetParticularPart(TestKeyManagerImpl.java:1567)

https://github.com/apache/ozone/actions/runs/12687113055/job/35361805439?pr=7558#step:6:4472

[juncevich]

Let's wait for a clean CI.

Please check failures:

Error:  Tests run: 29, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 197.004 s <<< FAILURE! - in org.apache.hadoop.ozone.om.TestKeyManagerImpl
Error:  org.apache.hadoop.ozone.om.TestKeyManagerImpl.testGetNotExistedPart  Time elapsed: 0.045 s  <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <0> but was: <5>
...
	at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:531)
	at org.apache.hadoop.ozone.om.TestKeyManagerImpl.testGetNotExistedPart(TestKeyManagerImpl.java:1591)

Error:  org.apache.hadoop.ozone.om.TestKeyManagerImpl.testGetParticularPart  Time elapsed: 0.039 s  <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <1> but was: <5>
...
	at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:531)
	at org.apache.hadoop.ozone.om.TestKeyManagerImpl.testGetParticularPart(TestKeyManagerImpl.java:1567)

https://github.com/apache/ozone/actions/runs/12687113055/job/35361805439?pr=7558#step:6:4472

@adoroszlai, thank you for your message. Build again green.

[adoroszlai]

Thanks @juncevich for the patch, @ivandika3 for the review.