List objects and common prefixes by carsonfarmer · Pull Request #45 · recallnet/ipc

carsonfarmer · 2024-03-25T22:59:48Z

Adds support for S3 style ObjectList API. The list method on the actor now takes multiple arguments, including prefix, delimiter, and limit. It Requires that these are BytesKeys, which are a type to be used to serialize as byte string instead of a u8 array. This allows us to assume byte string semantics for string comparisons etc.

Once merged, this provides the following additions to the /os proxy API. Listing is mostly as before, though the actual objects are now nested within the objects key (which is an array):

curl -s http://localhost:8001/v1/os | jq ".objects[].key"
"one|two.pdf"
"one"
"blah|bitcoin.pdf"
"one.pdf"

When delimiter is used, any common prefixes are aggregated into the common_prefixes key and included in the response:

curl -s http://localhost:8001/v1/os\?delimiter\=\| | jq             
{
  "common_prefixes": [
    "blah|",
    "one|"
  ],
  "objects": [
    {
      "key": "one",
      "resolved": false,
      "value": "bafk2bzacecslkk4nsiwdf5sx3i3xi6b6yj73uvli3p2u3bmczisougobh5bqs"
    },
    {
      "key": "one.pdf",
      "resolved": false,
      "value": "bafk2bzaceammiknpp4joitwsvtcc3u3ni47bvvbbaqgjed5xieraknhdxb6pc"
    }
  ]
}

One downside of the proxy API that we are using for the object store (at the moment), is that it isn't possible to push files with forward slashes as the delimiter... hense the | in the examples above.

Fixes #13

sanderpick · 2024-03-29T23:52:02Z

the failed test is due to some lint issues... you can do make lint locally before pushing

Signed-off-by: Carson Farmer <carson.farmer@gmail.com>

carsonfarmer · 2024-03-30T05:21:55Z

fendermint/actors/objectstore/src/state.rs

@@ -0,0 +1,384 @@
+// Copyright 2024 Textile


Seems silly, but I actually pulled this temporarily into a separate project and testing things out there. Then I found it was quite nice to have the state all self-contained like this, so I kept it.

carsonfarmer · 2024-03-30T05:22:32Z

fendermint/app/src/cmd/proxy.rs

 }

+#[derive(Serialize, Deserialize)]
+struct ListQuery {


Everything is optional on the web.

carsonfarmer · 2024-03-30T05:24:10Z

fendermint/app/src/cmd/proxy.rs

    hq: HeightQuery,
 ) -> Result<impl Reply, Rejection> {
-    let res = os_list(client, args, hq.height.unwrap_or(0))
+    let opts = ListOptions {


I opted to use the same ListOptions struct all the way up the way down to the actor state. It just keeps things simple. However, it would have been much nicer to use something more like the ListQuery structure, which contains Options. I'm not clear on whether we can use these with IPLD encoding/decoding, so I opted not to.

the ipld serializers do indeed handle Option, but this looks fine!

Ah ok, very nice. I should have just checked. I might make the switch to option at some point in the future then.

carsonfarmer · 2024-03-30T05:25:17Z

fendermint/actors/objectstore/src/state.rs

+    }
+}
+
+impl State {


All the existing methods are unchanged.

carsonfarmer · 2024-03-30T05:25:42Z

fendermint/actors/objectstore/src/state.rs

+}
+
+#[derive(Default, Debug, Serialize_tuple, Deserialize_tuple)]
+pub struct ObjectList {


This is a new struct, which is quite similar to the AWS ListObjects response object.

carsonfarmer · 2024-03-30T05:26:13Z

fendermint/actors/objectstore/src/state.rs

+        Ok(value)
+    }
+
+    pub fn list<BS: Blockstore>(


Here's the core change on the state.

carsonfarmer · 2024-03-30T05:26:47Z

fendermint/actors/objectstore/src/state.rs

+        let hamt = Hamt::<_, Object>::load_with_bit_width(&self.root, store, BIT_WIDTH)?;
+        let mut objects = Vec::new();
+        let mut common_prefixes = std::collections::BTreeSet::<Vec<u8>>::new();
+        let limit = if options.limit == 0 {


If the limit is zero, we make it the max, but anything else we "crop" to the max without explanation.

carsonfarmer · 2024-03-30T05:27:38Z

fendermint/actors/objectstore/src/state.rs

+            (options.limit as usize).min(MAX_LIST_LIMIT)
+        };
+        let offset = options.offset;
+        let mut count = 0;


This is kinda lame, but since our hamt doesn't provide a way to directly offset into the key space, we need to count manually.

carsonfarmer · 2024-03-30T05:28:44Z

fendermint/actors/objectstore/src/state.rs

+        for pair in &hamt {
+            let (k, v) = pair?;
+            let key = k.0.clone();
+            if !options.prefix.is_empty() && !key.starts_with(&options.prefix) {


skip this one if we have a prefix and it doesn't match. note that here, we're doing direct byte comparisons (faster), whereas below, when doing the delimiter matching, we need to convert to utf8 strings...

fendermint/actors/objectstore/src/state.rs

Signed-off-by: Carson Farmer <carson.farmer@gmail.com>

carsonfarmer

Added support for nesting prefixes/delimiters.

carsonfarmer · 2024-03-30T06:04:59Z

fendermint/actors/objectstore/src/state.rs

+                let utf8_key = String::from_utf8(key.clone())?;
+                let utf8_delimiter = String::from_utf8(options.delimiter.clone())?;
+                if let Some(index) = utf8_key[prefix_length..].find(&utf8_delimiter) {
+                    let subset = utf8_key[..=(index + prefix_length)].as_bytes().to_owned();


When listing common prefixes, we include the delimiter itself, and the full path up to the delimiter.

carsonfarmer · 2024-03-30T06:06:24Z

fendermint/actors/objectstore/src/state.rs

+                let prefix_length = utf8_prefix.len();
+                let utf8_key = String::from_utf8(key.clone())?;
+                let utf8_delimiter = String::from_utf8(options.delimiter.clone())?;
+                if let Some(index) = utf8_key[prefix_length..].find(&utf8_delimiter) {


We only search for the delimiter after the prefix. This allows us to essentially do nested prefixes and a folder-like hierarchy. But, we always include the full path up to the first delimiter after the prefix. I'm sure there is a cleaner way to do this.

carsonfarmer · 2024-03-30T06:08:05Z

the failed test is due to some lint issues... you can do make lint locally before pushing

Actually, I ended up just changing my "check" command in VSCode from "check" to "clippy". So this is all automatically checked now. But a nice side effect of this is that clippy appears to be way faster than the default "check"?

sanderpick

great work!

sanderpick · 2024-03-30T17:44:01Z

fendermint/app/src/cmd/proxy.rs

    hq: HeightQuery,
 ) -> Result<impl Reply, Rejection> {
-    let res = os_list(client, args, hq.height.unwrap_or(0))
+    let opts = ListOptions {


the ipld serializers do indeed handle Option, but this looks fine!

sanderpick · 2024-03-30T17:58:57Z

in the PR description, in the delimiter example, if I understand common_prefixes properly, the actual keys present are:

blah|one|one
blah|one|one.pdf

is that right?

carsonfarmer · 2024-03-30T18:04:12Z

Nope, the actual keys are those listed in the prior example. It lists all keys that match a given prefix up to and including the delimiter. Note in the second example with delimiter there is no prefix, so it lists anything up to the first delimiter as a common prefix.

Think of it like listing all files in the folder at the given prefix on down, but if a delimiter is supplied, then stop there, essentially listing sub folders but no lower.

sanderpick · 2024-03-30T18:08:02Z

ah! slick, makes sense 🙏

sanderpick · 2024-03-30T18:29:32Z

mind if I merge this?

depends on [entangler #45](recallnet/entanglement#45), after merging I'll update version deps to match the new version on engangler/main. targets #625, this updates iroh to version 0.35, which will match the version of the deployed relays. This PR also incorporates the streaming updates & config constant change that drops the `ENTANGLER_P` param, which should be reviewed.

carsonfarmer added the enhancement New feature or request label Mar 25, 2024

carsonfarmer added this to the Usable SDK/CLI milestone Mar 25, 2024

carsonfarmer self-assigned this Mar 25, 2024

carsonfarmer force-pushed the carson/prefixes branch from 9544a61 to 65b79ff Compare March 25, 2024 23:02

carsonfarmer marked this pull request as ready for review March 26, 2024 19:18

carsonfarmer added 6 commits March 29, 2024 21:49

chore: add some missing items to gitignore

245d477

Signed-off-by: Carson Farmer <carson.farmer@gmail.com>

feat: tooling to list objs and common prefixes

7bcd2c6

Signed-off-by: Carson Farmer <carson.farmer@gmail.com>

feat: wire it all up

6a4d13c

Signed-off-by: Carson Farmer <carson.farmer@gmail.com>

fix: compiler fixes and better api

fd9d052

Signed-off-by: Carson Farmer <carson.farmer@gmail.com>

feat: return object rather than list

72ac015

Signed-off-by: Carson Farmer <carson.farmer@gmail.com>

feat: add an offset and condense opts into struct

aaa2686

Signed-off-by: Carson Farmer <carson.farmer@gmail.com>

carsonfarmer force-pushed the carson/prefixes branch from dc9cd48 to aaa2686 Compare March 30, 2024 04:50

carsonfarmer changed the base branch from develop to main March 30, 2024 05:20

carsonfarmer changed the base branch from main to develop March 30, 2024 05:20

carsonfarmer commented Mar 30, 2024

View reviewed changes

feat: support for prefixes with delimiters for nesting

e15d398

Signed-off-by: Carson Farmer <carson.farmer@gmail.com>

carsonfarmer commented Mar 30, 2024

View reviewed changes

sanderpick approved these changes Mar 30, 2024

View reviewed changes

carsonfarmer merged commit fc1e926 into develop Mar 31, 2024

carsonfarmer deleted the carson/prefixes branch March 31, 2024 02:43

Conversation

carsonfarmer commented Mar 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sanderpick commented Mar 29, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

carsonfarmer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carsonfarmer commented Mar 30, 2024

Uh oh!

sanderpick left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sanderpick commented Mar 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carsonfarmer commented Mar 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sanderpick commented Mar 30, 2024

Uh oh!

sanderpick commented Mar 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

carsonfarmer commented Mar 25, 2024 •

edited

Loading

sanderpick commented Mar 30, 2024 •

edited

Loading

carsonfarmer commented Mar 30, 2024 •

edited

Loading