Skip to content

Required optional ETag on .list() #3477

@amunra

Description

@amunra

I'm developing against s3s-fs and using OpenDAL to read the data.

It looks like listing currently expects an <ETag> to be set, despite this being marked as optional in the Amazon docs (https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.html).

The following diff seems to work around the problem.

I'll try and raise a proper PR for this tiny change, but I'd appreciate feedback in the meantime.

(I see @Xuanwo in the commit history for this part of the code, so tagging you - I hope you don't mind!).

diff --git a/core/src/services/s3/pager.rs b/core/src/services/s3/pager.rs
index 3c5b15ed0..30646da8a 100644
--- a/core/src/services/s3/pager.rs
+++ b/core/src/services/s3/pager.rs
@@ -126,8 +126,10 @@ impl oio::Page for S3Pager {
 
             let mut meta = Metadata::new(EntryMode::FILE);
 
-            meta.set_etag(&object.etag);
-            meta.set_content_md5(object.etag.trim_matches('"'));
+            if let Some(etag) = &object.etag {
+                meta.set_etag(&etag);
+                meta.set_content_md5(etag.trim_matches('"'));
+            }
             meta.set_content_length(object.size);
 
             // object.last_modified provides more precious time that contains
@@ -168,7 +170,7 @@ struct OutputContent {
     size: u64,
     last_modified: String,
     #[serde(rename = "ETag")]
-    etag: String,
+    etag: Option<String>,
 }
 
 #[derive(Default, Debug, Eq, PartialEq, Deserialize)]

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions