Skip to content

[Feature]: Add disable_batch_delete option to support Alibaba Cloud OSS in S3 backend #1330

@yuan7he

Description

@yuan7he

Problem Statement

When using Alibaba Cloud OSS as the AGFS S3 backend, add_resource always fails with a 400 error during temporary directory cleanup after file processing:

failed to delete source directory: failed to delete objects:
operation error S3: DeleteObjects, StatusCode: 400,
api error MissingArgument: Missing Some Required Arguments.

The missing argument is Content-MD5. OSS requires this header for the DeleteObjects (POST /?delete) API, but AWS SDK v2 does not send it by default. This
makes the S3 backend completely unusable with Alibaba Cloud OSS.

Proposed Solution

Add a disable_batch_delete boolean option to S3Config. When set to true, the backend uses sequential single-object DeleteObject calls instead of batch
DeleteObjects. Single-object delete does not require Content-MD5, making it compatible with OSS.

The option defaults to false, so existing users are not affected.

Example configuration:

"s3": {
  "endpoint": "http://s3.oss-cn-beijing.aliyuncs.com",
  "bucket": "your-bucket",
  "region": "cn-beijing",
  "disable_batch_delete": true
}

Use Case

Using Alibaba Cloud OSS (s3.oss-{region}.aliyuncs.com) as the AGFS S3 storage backend. OSS is a widely used object storage service that follows the
S3-compatible API with minor differences — one being the mandatory Content-MD5 header for batch delete operations.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions