HDDS-11784. Add missing parent directories for abort MPU and abort expired MPU requests

[sokui]

What changes were proposed in this pull request?

HDDS-11784 adding missing parent directories for MPU abort and expired abort request

We observed lots of open key (files) in our FSO enabled ozone cluster. And these are all incomplete MPU keys.

When I tried to abort MPU by using s3 cli as below, I got the exception complaining about the parent directory is not found.

aws s3api abort-multipart-upload --endpoint 'xxxx' --bucket '2e76bd0f-9682-42c6-a5ce-3e32c5aa37b2' --key 'CACHE.06e656c0-6622-48bb-89c2-39470764b1d0/abc.blob' --upload-id '4103c881-24fa-4992-b7b2-5474f8a7fbaf-113204926929050074'

An error occurred (NoSuchUpload) when calling the AbortMultipartUpload operation: The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.

Exceptions in the log

NO_SUCH_MULTIPART_UPLOAD_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Abort Multipart Upload Failed: volume: s3v, bucket: 2e76bd0f-9682-42c6-a5ce-3e32c5aa37b2, key: CACHE.06e656c0-6622-48bb-89c2-39470764b1d0/abc.blob
at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadAbortRequest.validateAndUpdateCache(S3MultipartUploadAbortRequest.java:148)
at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.lambda$0(OzoneManagerRequestHandler.java:402)
at org.apache.hadoop.util.MetricUtil.captureLatencyNs(MetricUtil.java:39)
at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:398)
at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:587)
at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$1(OzoneManagerStateMachine.java:375)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: DIRECTORY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Failed to find parent directory of CACHE.06e656c0-6622-48bb-89c2-39470764b1d0/abc.blob
at org.apache.hadoop.ozone.om.request.file.OMFileRequest.getParentID(OMFileRequest.java:1038)
at org.apache.hadoop.ozone.om.request.file.OMFileRequest.getParentID(OMFileRequest.java:988)
at org.apache.hadoop.ozone.om.request.util.OMMultipartUploadUtils.getMultipartOpenKeyFSO(OMMultipartUploadUtils.java:122)
at org.apache.hadoop.ozone.om.request.util.OMMultipartUploadUtils.getMultipartOpenKey(OMMultipartUploadUtils.java:99)
at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadAbortRequest.getMultipartOpenKey(S3MultipartUploadAbortRequest.java:256)
at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadAbortRequest.validateAndUpdateCache(S3MultipartUploadAbortRequest.java:145)
... 9 more

This issue is similar as the issue HDDS-10630. This PR brings the same mechanism which adds the missing parent directories to the table cache before we abort the MPU. The PR adds this mechanism to both S3ExpiredMultipartUploadsAbortRequest and S3ExpiredMultipartUploadsAbortRequest. Also the code is refactored so multiple places reuse the same mechanism

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11784

How was this patch tested?

It tested by the CI. Also we validate it in our cluster (on going).

[adoroszlai]

It tested by the CI

This is usually expected to mean "CI completed successfully with these changes". However, in this case, as you have noticed, some tests are failing:

https://github.com/sokui/ozone/actions/runs/12286612359/job/34288154427#step:6:759

Unfortunately CI does not provide logs, because individual test cases after the failing one keep timing out, eventually reaching overall CI timeout. I'll make a note to improve that.

You can run the same test locally.

First, reduce test cases to minimum:

1. Edit hadoop-ozone/dist/src/main/compose/ozone-ha/test.sh:
   • delete everything after start_docker_env 5
   • add execute_robot_test ${SCM} -v BUCKET_LAYOUT:FILE_SYSTEM_OPTIMIZED s3/MultipartUpload.robot
2. Edit hadoop-ozone/dist/src/main/smoketest/s3/MultipartUpload.robot
   • delete all test cases after Test abort Multipart upload with invalid uploadId

mvn -DskipTests -DskipShade clean package
cd hadoop-ozone/dist/target/ozone-2.0.0-SNAPSHOT/compose/ozone-ha
./test.sh

I tried this locally, it seems OM terminates when aborting unknown MPU. Can you please check?

[sokui]

I fixed the test error. But the new CI run has another test failure org.apache.hadoop.fs.ozone.TestHSync . However, when I run this test locally, it has no issues. I am not sure if this test is already flaky. See the CI run on my repo here

[adoroszlai]

Thanks @sokui for fixing the test, will take a look. Don't worry about TestHSync, it's unrelated and flaky.

[adoroszlai]

@sokui Please try to avoid force-push when updating the PR. Here are some great articles that explain why:

https://developers.mattermost.com/blog/submitting-great-prs/#4-avoid-force-pushing
https://www.freecodecamp.org/news/optimize-pull-requests-for-reviewer-happiness#request-a-review

[sokui]

@sokui Please try to avoid force-push when updating the PR. Here are some great articles that explain why:

https://developers.mattermost.com/blog/submitting-great-prs/#4-avoid-force-pushing https://www.freecodecamp.org/news/optimize-pull-requests-for-reviewer-happiness#request-a-review

Sure thing. I was try to cherry pick my change as one commit and put it above 1.4.1, and use it as our internal build for testing. That is why I do the force push. I will not force push for my future commits. Thanks for reminder.

[adoroszlai]

Thanks @sokui for the patch.

[sokui]

@adoroszlai pls have another look. Thank you!

[ivandika3]

@SaketaChalamchala Could you help to take a look as well since it's related to the logic in HDDS-10630?

[ivandika3]

@sokui Thanks for working on this. Just an initial review as I'm trying to understand the related logic.

Also I noticed in #6496 that the DIRECTORY_TABLE is not added in the @CleanupTableInfo in the related OM response. Please help to add the DIRECTORY_TABLE for the OM responses which add the missing directories to the directory table.

Could you also help to add some tests? I'm not entirely sure in what conditions will missing parent directories might happen.

[sokui]

@sokui Thanks for working on this. Just an initial review as I'm trying to understand the related logic.

Also I noticed in #6496 that the DIRECTORY_TABLE is not added in the @CleanupTableInfo in the related OM response. Please help to add the DIRECTORY_TABLE for the OM responses which add the missing directories to the directory table.

Could you also help to add some tests? I'm not entirely sure in what conditions will missing parent directories might happen.

Hi @ivandika3 ,

Correct me if I'm wrong. Here, we only add the missing directories to the cache of the directory table (since we need to delete this key), not the real directory table. In this case, do we need to add DIRECTORY_TABLE to @CleanupTableInfo?

[ivandika3]

Correct me if I'm wrong. Here, we only add the missing directories to the cache of the directory table (since we need to delete this key), not the real directory table. In this case, do we need to add DIRECTORY_TABLE to @CleanupTableInfo?

At least in #6496, since the OM response will add the missing directories to the real directory table, @CleanupTableInfo should be added to clean the OM directory cache entries since they are already persisted in OM DB.

As for abort and expired abort MPU. I think it's OK to just add to the dir cache since the directories are just to avoid the exceptions. However, IMO we should also add the @CleanupTableInfo for the corresponding OM response so that the cache entries will not be in the OM dir cache forever.

That said, AFAIK this is the only special case where OM request update to cache does not translate to persisting the entry in the OM DB. Hence, I'm not entire sure what might be the possible impact. For example, one case I can think of is that if the subsequent key is created using that missing directories created by this patch, but later when the cache is cleaned up or OM is restarted, the cached directories disappear and the newly created key will be orphaned.

[sokui]

Could you also help to add some tests? I'm not entirely sure in what conditions will missing parent directories might happen.

Hi @ivandika3 , to be honest, I am not sure what the root cause. As stated in #6496, the files (S3InitiateMultipartUploadRequestWithFSO.java, S3InitiateMultipartUploadResponseWithFSO.java) contains this logic.I can understand we are missing the parent directories initially, but why we missing these directories in the complete, abort and exprire phases? Is it possible that between adding the open key to the table and adding the missing parent directories, ozone (om) got interrupted, leaving the open key exists but parent directories not there?

[ivandika3]

@sokui Ok, I was thinking a bit more about this.

why we missing these directories in the complete, abort and exprire phases?

I can think one possible case, when between the initiate and complete / abort / expire, the parent directory is deleted and all the child directories and files are garbage collected by DirectoryDeletingService (runs every 60s).

For example, there is a directory "/a" and there is a pending MPU key with path "/a/mpu_key" that was initiated but haven't been completed / aborted yet. After the MPU key was initiated, the directory "/a" is deleted, and since mpu_key has not been completed yet and does not exist in fileTable, the DIRECTORY_NOT_EMPTY will not be thrown in OMKeyDeleteRequestWithFSO#validateAndUpdateCache. Therefore mpu_key is orphaned and when it's completed / aborted, it will fail in OMFileRequest#getParentID since the parent directory has been deleted.

Now I can understand that initiate and complete MPU will need to add the missing parent directories, since the directories will be accessed again. However, for abort (expired) MPUs, I don't think #6496 applies since we don't really want to add the missing parent directories for a key that is going to be deleted anyway. Even if we add the parent directories, adding it in cache only might result in unpredictable behavior where the missing directories created can suddenly disappear since they are not persisted in the OM DB. So, I believe either we should persist the missing parent directories or we should not create them in the first place.

I think the original intention is that we want to ensure the MPU abort will succeed regardless whether the parent exists or not. I can think of one way, how about we first get the OmMultipartKeyInfo from the multipartInfoTable since it does not require parentId information. Afterwards, we use the OmMultipartKeyInfo#getParentId to derive the MPU open key, without needing to call OMFileRequest#getParentId. That way, we don't need to create the missing parent directories.

In the future, for preventive measures, we can try to take into account incomplete MPUs for directory deletions. For example:

• OMFileRequest#hasChildren might need to return true if there is a incomplete MPU key underneath it.
• DirectoryDeletingService can also add some logic to abort the incomplete MPU
  However, these can be done in the future tickets.

Note: I'm not too well-versed about FSO logic, so I could be wrong.

[sokui]

@sokui Ok, I was thinking a bit more about this.

why we missing these directories in the complete, abort and exprire phases?

I can think one possible case, when between the initiate and complete / abort / expire, the parent directory is deleted and all the child directories and files are garbage collected by DirectoryDeletingService (runs every 60s).

For example, there is a directory "/a" and there is a pending MPU key with path "/a/mpu_key" that was initiated but haven't been completed / aborted yet. After the MPU key was initiated, the directory "/a" is deleted, and since mpu_key has not been completed yet and does not exist in fileTable, the DIRECTORY_NOT_EMPTY will not be thrown in OMKeyDeleteRequestWithFSO#validateAndUpdateCache. Therefore mpu_key is orphaned and when it's completed / aborted, it will fail in OMFileRequest#getParentID since the parent directory has been deleted.

Now I can understand that initiate and complete MPU will need to add the missing parent directories, since the directories will be accessed again. However, for abort (expired) MPUs, I don't think #6496 applies since we don't really want to add the missing parent directories for a key that is going to be deleted anyway. Even if we add the parent directories, adding it in cache only might result in unpredictable behavior where the missing directories created can suddenly disappear since they are not persisted in the OM DB. So, I believe either we should persist the missing parent directories or we should not create them in the first place.

I think the original intention is that we want to ensure the MPU abort will succeed regardless whether the parent exists or not. I can think of one way, how about we first get the OmMultipartKeyInfo from the multipartInfoTable since it does not require parentId information. Afterwards, we use the OmMultipartKeyInfo#getParentId to derive the MPU open key, without needing to call OMFileRequest#getParentId. That way, we don't need to create the missing parent directories.

In the future, for preventive measures, we can try to take into account incomplete MPUs for directory deletions. For example:

• OMFileRequest#hasChildren might need to return true if there is a incomplete MPU key underneath it.
• DirectoryDeletingService can also add some logic to abort the incomplete MPU
  However, these can be done in the future tickets.

Note: I'm not too well-versed about FSO logic, so I could be wrong.

I am testing your idea. Will keep you updated.

[sokui]

I can think of one way, how about we first get the OmMultipartKeyInfo from the multipartInfoTable since it does not require parentId information. Afterwards, we use the OmMultipartKeyInfo#getParentId to derive the MPU open key, without needing to call OMFileRequest#getParentId. That way, we don't need to create the missing parent directories.

@ivandika3 verified. Your idea works. I will modify my PR accordingly.

[sokui]

I created a new PR with the new idea here. Pls review that PR. Will close this one if the other one is good.

[adoroszlai]

Thanks @sokui for continuing work on this. I think it's OK to completely update PR based on comments if a better approach is found. Creating new one splits discussion between the two PRs. Now that the new PR is open, let's close this one.

[sokui]

Thanks @sokui for continuing work on this. I think it's OK to completely update PR based on comments if a better approach is found. Creating new one splits discussion between the two PRs. Now that the new PR is open, let's close this one.

I understand your consideration. I have two considerations if I continue working on the same PR: first the other approach is very simple, if I continue work on this PR, I need to revert each of the comment which looks weird. Second, I am not sure at that time point which direction should go. Thats why I create a new PR without touching this one. In case we still favor this one, I can go back without any impact.

To let people know our conversation, I will link this PR in the other PR.