HDDS-11784. Allow aborting FSO multipart uploads with missing parent directories

[sokui]

What changes were proposed in this pull request?

HDDS-11784 adding missing parent directories for MPU abort and expired abort request

We observed lots of open key (files) in our FSO enabled ozone cluster. And these are all incomplete MPU keys.

When I tried to abort MPU by using s3 cli as below, I got the exception complaining about the parent directory is not found.

aws s3api abort-multipart-upload --endpoint 'xxxx' --bucket '2e76bd0f-9682-42c6-a5ce-3e32c5aa37b2' --key 'CACHE.06e656c0-6622-48bb-89c2-39470764b1d0/abc.blob' --upload-id '4103c881-24fa-4992-b7b2-5474f8a7fbaf-113204926929050074'

An error occurred (NoSuchUpload) when calling the AbortMultipartUpload operation: The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.

Exceptions in the log

NO_SUCH_MULTIPART_UPLOAD_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Abort Multipart Upload Failed: volume: s3v, bucket: 2e76bd0f-9682-42c6-a5ce-3e32c5aa37b2, key: CACHE.06e656c0-6622-48bb-89c2-39470764b1d0/abc.blob
at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadAbortRequest.validateAndUpdateCache(S3MultipartUploadAbortRequest.java:148)
at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.lambda$0(OzoneManagerRequestHandler.java:402)
at org.apache.hadoop.util.MetricUtil.captureLatencyNs(MetricUtil.java:39)
at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:398)
at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:587)
at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$1(OzoneManagerStateMachine.java:375)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: DIRECTORY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Failed to find parent directory of CACHE.06e656c0-6622-48bb-89c2-39470764b1d0/abc.blob
at org.apache.hadoop.ozone.om.request.file.OMFileRequest.getParentID(OMFileRequest.java:1038)
at org.apache.hadoop.ozone.om.request.file.OMFileRequest.getParentID(OMFileRequest.java:988)
at org.apache.hadoop.ozone.om.request.util.OMMultipartUploadUtils.getMultipartOpenKeyFSO(OMMultipartUploadUtils.java:122)
at org.apache.hadoop.ozone.om.request.util.OMMultipartUploadUtils.getMultipartOpenKey(OMMultipartUploadUtils.java:99)
at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadAbortRequest.getMultipartOpenKey(S3MultipartUploadAbortRequest.java:256)
at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadAbortRequest.validateAndUpdateCache(S3MultipartUploadAbortRequest.java:145)
... 9 more

This issue is because of missing the parent directories. The causation and the solution is explained here: #7566 (comment).

There is another PR which we closed. We had some conversation about which approach should be taken. Pls reference here: #7566

I think the original intention is that we want to ensure the MPU abort will succeed regardless whether the parent exists or not. I can think of one way, how about we first get the OmMultipartKeyInfo from the multipartInfoTable since it does not require parentId information. Afterwards, we use the OmMultipartKeyInfo#getParentId to derive the MPU open key, without needing to call OMFileRequest#getParentId. That way, we don't need to create the missing parent directories.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11784

How was this patch tested?

It tested by the CI. Also we validate it in our cluster.

[ivandika3]

@sokui Thanks for the patch.

Please check the test failures

• https://github.com/sokui/ozone/actions/runs/12783047651/job/35633751006

getMultipartKeyFSO is used by a lot of MPU flows. Furthermore, multipartInfoTable will be accessed two times although. I think we can create another function similar to getMultipartKeyFSO by passing the OmMultipartKeyInfo and use that only for the abort case. This means switching the order in S3MultipartAbortRequest by first getting from multipartInfoTable and then to openFileTable. All other implementations are welcome.

Also let's add a simple test as suggested in #7566 (comment)

For example, there is a directory "/a" and there is a pending MPU key with path "/a/mpu_key" that was initiated but haven't been completed / aborted yet. After the MPU key was initiated, the directory "/a" is deleted, and since mpu_key has not been completed yet and does not exist in fileTable, the DIRECTORY_NOT_EMPTY will not be thrown in OMKeyDeleteRequestWithFSO#validateAndUpdateCache. Therefore mpu_key is orphaned and when it's completed / aborted, it will fail in OMFileRequest#getParentID since the parent directory has been deleted.

[sokui]

@sokui Thanks for the patch.

Please check the test failures

• https://github.com/sokui/ozone/actions/runs/12783047651/job/35633751006

getMultipartKeyFSO is used by a lot of MPU flows. Furthermore, multipartInfoTable will be accessed two times although. I think we can create another function similar to getMultipartKeyFSO by passing the OmMultipartKeyInfo and use that only for the abort case. This means switching the order in S3MultipartAbortRequest by first getting from multipartInfoTable and then to openFileTable. All other implementations are welcome.

Also let's add a simple test as suggested in #7566 (comment)

For example, there is a directory "/a" and there is a pending MPU key with path "/a/mpu_key" that was initiated but haven't been completed / aborted yet. After the MPU key was initiated, the directory "/a" is deleted, and since mpu_key has not been completed yet and does not exist in fileTable, the DIRECTORY_NOT_EMPTY will not be thrown in OMKeyDeleteRequestWithFSO#validateAndUpdateCache. Therefore mpu_key is orphaned and when it's completed / aborted, it will fail in OMFileRequest#getParentID since the parent directory has been deleted.

My consideration is that org.apache.hadoop.ozone.om.request.util.OMMultipartUploadUtils#getMultipartOpenKey is used by multiple places, including S3ExpiredMultipartUploadsAbortRequest and S3MultipartUploadAbortRequest. By updating one place here, it benefits for two places (both not work currently). Secondly, if my current implementation of getMultipartKeyFSO is more reliable, there is no reason to only limit this benefit to S3MultipartUploadAbortRequest. All the other places should use this as well.

I originally implemented the exact way you describe. But considering the above reasons, I change to directly modify getMultipartKeyFSO. Pls let me know if it makes sense.

For the interface, I noticed OmMultipartKeyInfo is already there in S3MultipartUploadAbortRequest. Let me take a look how to make the interface better so that it can be used in all places.

For the tests, I will take a look the failure. For the new test you suggested, do you know if there is similar testing existing so that I can reference it? I am not super familiar with ozone code base. So if there is no such similar code there, could you pls show me some code snippet which I can start with. Really appreciate it!

[ivandika3]

My consideration is that org.apache.hadoop.ozone.om.request.util.OMMultipartUploadUtils#getMultipartOpenKey is used by multiple places, including S3ExpiredMultipartUploadsAbortRequest and S3MultipartUploadAbortRequest. By updating one place here, it benefits for two places (both not work currently). Secondly, if my current implementation of getMultipartKeyFSO is more reliable, there is no reason to only limit this benefit to S3MultipartUploadAbortRequest. All the other places should use this as well.

My worry is that there might be some places where the expectations for OMMultipartUploadUtils#getMultipartKeyFSO is to access the open key/file table or there might some places where the multipartInfoTable entry does not exist yet, which might result in NPE which might crash the OM (we should handle the possible NPE). However, I'm OK as long as there are no test regressions.

For the tests, I will take a look the failure. For the new test you suggested, do you know if there is similar testing existing so that I can reference it? I am not super familiar with ozone code base. So if there is no such similar code there, could you pls show me some code snippet which I can start with. Really appreciate it!

You can start with TestOzoneClientMultipartUploadWithFSO integration test.

[sokui]

My consideration is that org.apache.hadoop.ozone.om.request.util.OMMultipartUploadUtils#getMultipartOpenKey is used by multiple places, including S3ExpiredMultipartUploadsAbortRequest and S3MultipartUploadAbortRequest. By updating one place here, it benefits for two places (both not work currently). Secondly, if my current implementation of getMultipartKeyFSO is more reliable, there is no reason to only limit this benefit to S3MultipartUploadAbortRequest. All the other places should use this as well.

My worry is that there might be some places where the expectations for OMMultipartUploadUtils#getMultipartKeyFSO is to access the open key/file table or there might some places where the multipartInfoTable entry does not exist yet, which might result in NPE which might crash the OM (we should handle the possible NPE). However, I'm OK as long as there are no test regressions.

For the tests, I will take a look the failure. For the new test you suggested, do you know if there is similar testing existing so that I can reference it? I am not super familiar with ozone code base. So if there is no such similar code there, could you pls show me some code snippet which I can start with. Really appreciate it!

You can start with TestOzoneClientMultipartUploadWithFSO integration test.

I started with some test with the following code. But It seem I cannot delete the directory.

    String parentDir = "a/b";
    keyName = parentDir + UUID.randomUUID();

    OzoneManager ozoneManager = cluster.getOzoneManager();
    String buckKey = ozoneManager.getMetadataManager()
        .getBucketKey(volume.getName(), bucket.getName());
    OmBucketInfo buckInfo =
        ozoneManager.getMetadataManager().getBucketTable().get(buckKey);
    BucketLayout bucketLayout = buckInfo.getBucketLayout();

    String uploadID = initiateMultipartUploadWithAsserts(bucket, keyName, RATIS,
        ONE);
    bucket.deleteDirectory(parentDir, false);

It gave me the following error:

KEY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Unable to get file status: volume: 8f75be75-539f-4fb1-8cc0-8c1123f1710f bucket: 2ce72254-3315-4662-95ae-cd09132ef932 key: a/b

	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:763)
	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.deleteKey(OzoneManagerProtocolClientSideTranslatorPB.java:962)
	at org.apache.hadoop.ozone.client.rpc.RpcClient.deleteKey(RpcClient.java:1631)
	at org.apache.hadoop.ozone.client.OzoneBucket.deleteDirectory(OzoneBucket.java:689)
	at org.apache.hadoop.ozone.client.rpc.TestOzoneClientMultipartUploadWithFSO.testAbortUploadSuccessWithMissingParentDirectories(TestOzoneClientMultipartUploadWithFSO.java:648)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)

I checked the bucket.deleteDirectory() method. It looks like this:

public void deleteDirectory(String key, boolean recursive)
      throws IOException {
    proxy.deleteKey(volumeName, name, key, recursive);
  }

Just wonder if it is deleting a key or a directory? And if it is directory, why I got the above exception?

[ivandika3]

I started with some test with the following code. But It seem I cannot delete the directory.

@sokui The parentDir should be "a/b/" with trailing slash. Because there was no trailing slash, your key is "a/b<key_name>", so only "a/" parent directory is created. Therefore, deleting "a/b" will return KEY_NOT_FOUND (as it should be).

I created the following test, you can adapt it to your test.

  @Test
  public void testAbortMultipartUploadSuccessWithMissingParentDirectories() throws Exception {
    String parentDir = "parentDirToDelete/";
    keyName = parentDir + UUID.randomUUID();

    OzoneManager ozoneManager = cluster.getOzoneManager();
    String buckKey = ozoneManager.getMetadataManager()
        .getBucketKey(volume.getName(), bucket.getName());
    OmBucketInfo buckInfo =
        ozoneManager.getMetadataManager().getBucketTable().get(buckKey);
    BucketLayout bucketLayout = buckInfo.getBucketLayout();

    ozClient.getProxy().createDirectory(volumeName, bucketName, parentDir);

    String uploadID = initiateMultipartUpload(bucket, keyName, RATIS,
        ONE);
    Pair<String, String> partNameAndETag = uploadPart(bucket, keyName, uploadID,
        1, "data".getBytes(UTF_8));

    OMMetadataManager metadataMgr =
        cluster.getOzoneManager().getMetadataManager();
    String multipartKey = verifyUploadedPart(uploadID, partNameAndETag.getKey(),
        metadataMgr);

    // Delete parent directory
    ozClient.getProxy().deleteKey(volumeName, bucketName, parentDir, false);

    // Abort multipart upload with missing parent directory
    bucket.abortMultipartUpload(keyName, uploadID);

    String multipartOpenKey =
        metadataMgr.getMultipartKeyFSO(volumeName, bucketName, keyName, uploadID);
    OmKeyInfo omKeyInfo =
        metadataMgr.getOpenKeyTable(bucketLayout).get(multipartOpenKey);
    OmMultipartKeyInfo omMultipartKeyInfo =
        metadataMgr.getMultipartInfoTable().get(multipartKey);
    assertNull(omKeyInfo);
    assertNull(omMultipartKeyInfo);
  }

[sokui]

I started with some test with the following code. But It seem I cannot delete the directory.

@sokui The parentDir should be "a/b/" with trailing slash. Because there was no trailing slash, your key is "a/b<key_name>", so only "a/" parent directory is created. Therefore, deleting "a/b" will return KEY_NOT_FOUND (as it should be).

I created the following test, you can adapt it to your test.

  @Test
  public void testAbortMultipartUploadSuccessWithMissingParentDirectories() throws Exception {
    String parentDir = "parentDirToDelete/";
    keyName = parentDir + UUID.randomUUID();

    OzoneManager ozoneManager = cluster.getOzoneManager();
    String buckKey = ozoneManager.getMetadataManager()
        .getBucketKey(volume.getName(), bucket.getName());
    OmBucketInfo buckInfo =
        ozoneManager.getMetadataManager().getBucketTable().get(buckKey);
    BucketLayout bucketLayout = buckInfo.getBucketLayout();

    ozClient.getProxy().createDirectory(volumeName, bucketName, parentDir);

    String uploadID = initiateMultipartUpload(bucket, keyName, RATIS,
        ONE);
    Pair<String, String> partNameAndETag = uploadPart(bucket, keyName, uploadID,
        1, "data".getBytes(UTF_8));

    OMMetadataManager metadataMgr =
        cluster.getOzoneManager().getMetadataManager();
    String multipartKey = verifyUploadedPart(uploadID, partNameAndETag.getKey(),
        metadataMgr);

    // Delete parent directory
    ozClient.getProxy().deleteKey(volumeName, bucketName, parentDir, false);

    // Abort multipart upload with missing parent directory
    bucket.abortMultipartUpload(keyName, uploadID);

    String multipartOpenKey =
        metadataMgr.getMultipartKeyFSO(volumeName, bucketName, keyName, uploadID);
    OmKeyInfo omKeyInfo =
        metadataMgr.getOpenKeyTable(bucketLayout).get(multipartOpenKey);
    OmMultipartKeyInfo omMultipartKeyInfo =
        metadataMgr.getMultipartInfoTable().get(multipartKey);
    assertNull(omKeyInfo);
    assertNull(omMultipartKeyInfo);
  }

oops. I missed the ending /. Now it works. Thanks for pointing it out.

[sokui]

@ivandika3 addressed all the comments. Pls have another look. Thank you!

[kerneltime]

Overall we might want to revisit the multipart open key table key name calculation as a whole. I am not sure if the key should be based on normal FSO keys in the mutipart upload scenario. Maybe, we should just use the path of S3 key as is in the key calculation and not depend on the path/directories created? This fix might be ok for now but something to reconsider as a whole. I will go over the MPU code a few more times to complete the review.

[ivandika3]

Thanks for the update. LGTM +1.

However, please also address any suggestions from @kerneltime first.

[adoroszlai]

@kerneltime would you like to take another look?

[ivandika3]

@adoroszlai IMO this is good to go. We can address follow up tasks in another tickets.

[adoroszlai]

Thanks @sokui for the patch, @ivandika3 for the review.

[kerneltime]

@adoroszlai IMO this is good to go. We can address follow up tasks in another tickets.

I agree, I think we need to revisit how we write open keys for FSO buckets. This is not limited to MPU.