New API & transaction type for user to set new base paths

[jackye1995]

We have a new multi-bucket features that would benefit from the multiple base paths feature we added in shallow clone: #4765

The proposed API is something like:

...

dataset.add_base("s3://bucket/abc", name="primary")
dataset.add_base("s3://bucket3/abc", name="us-west-2")
dataset.add_base("s3://bucket3/abc", name="eu-west-2")

...

lance.write_dataset(data, dataset, mode="append", base_paths=["primary", "us-west-2"])

Or adding that as a part of dataset creation:

lance.write_dataset(data, dataset, mode="create", base_paths=[
  { "path": "s3://bucket/abc", "name": "primary" }, 
  { "path": "s3://bucket3/abc", "name": "us-west-2" }
])

There are 2 topics not closed yet:

1. it seems like we should give a deterministic name to the table's root location since it is not recorded in the path but we still want to reference it in the base_paths. In the example above, the user does not want to write data to the root location but only the newly added bases. But there should be also cases that user want to write to it, and should be able to specify something like base_paths=["root", "primary", "us-west-2"]
2. should we create new transaction operations like UpdateBasePaths? It seems like we can also just add it to the UpdateConfig operation, but that might be overloading it.

Curious what people think, @jaystarshot @majin1102 @wjones127

[jaystarshot]

Since write is only in one fragment at a time, we can consider adding a target_path for append / overwrite modes

lance.write_dataset(data, dataset, mode="create", 
target_path =primary,
base_paths=[
  { "path": "s3://bucket/abc", "name": "primary" }, 
  { "path": "s3://bucket3/abc", "name": "us-west-2" }
])``

and

lance.write_dataset(data, dataset, mode="append", target_path="primary"])

[westonpace]

I don't think we can use UpdateConfig because this would need to conflict with any other write operation right? Those operations would likely be placing data files in the wrong location?

[jackye1995]

because this would need to conflict with any other write operation right?

If we are updating or deleting an existing base, then things gets more complicated. But for the majority of the use cases of just adding new ones, existing writes should not conflict since they are just going to use known base paths.

[jackye1995]

Since write is only in one fragment at a time

@jaystarshot I think the base_paths in my example is already the target ones. As you can see there are actually 3 registered ones and only 2 are used. And I don't think we can assume there will only be one fragment at a time, that changes depending on the write parameters.

[westonpace]

But for the majority of the use cases of just adding new ones, existing writes should not conflict since they are just going to use known base paths.

Ok, I agree that adding new base paths should be harmless.

should we create new transaction operations like UpdateBasePaths? It seems like we can also just add it to the UpdateConfig operation, but that might be overloading it.

My mental model has been that we need a new operation when we need new conflict detection semantics but I can understand there is potentially other value in having more distinct transactions (e.g. clearer historical auditing of what has happened to a dataset). What would be your motivation here?

[jackye1995]

I think having a new transaction operation makes more sense to me, as you said it's a new semantics. I mainly want to confirm, because we already have quite a few operations and some ended up being not used and overlapping in functionality, so want to think through if anyone feels we should reuse existing ones.

[majin1102]

should we create new transaction operations like UpdateBasePaths?

The concern of a new operation type like UpdateBasePaths as I can see now is the ACID issue. If the main scenarios we import a base path are happening when we want to append a bouch of files. The API guides us to import the path first and then write data. That might introduce some unnecessary complexity to handle. For example, if the path uses a random string, the ACID issue would be a trouble.

On the contrary, if we don't need to write data, is there any scenario we should import a base path before data manipulation? If not I would suggest to add a parameter(or params?) in Append which would import paths into base_paths and make the whole thing more transactional.

But there should be also cases that user want to write to it, and should be able to specify something like base_paths=["root", "primary", "us-west-2"]

Two options I think both reasonable:

1. base_paths itself might not need to know the root concept if we don't do any validation check and people can add the root path as any name they want, maybe like an alias of root. They can append files using base_paths or just using defaut root.
2. Don't allow root path added into base_paths and make the root_path as the built-in element in base_paths. Make the root name as reserved.

Option 1 is prefered to me since we don't need to document much on this and keeps the flexibility.

[majin1102]

dataset.add_base("s3://bucket/abc", name="primary")

I'm sorry maybe there is something I missed. Do we have other usecases to use this API except appending files? @jackye1995

[jackye1995]

base_paths itself might not need to know the root concept if we don't do any validation check and people can add the root path as any name they want, maybe like an alias of root. They can append files using base_paths or just using defaut root.

That's a good point, I like this so it makes the logic simple, if the user wants to specify specific bases they can just register root as one of the bases.

[jackye1995]

Do we have other usecases to use this API except appending files?

this does not append files, it just registers the base. The goal is to then add new parameter in the other write APIs to allow specifying specific bases that the user wants the new fragment data files to be written into

[majin1102]

Do we have other usecases to use this API except appending files?

this does not append files, it just registers the base. The goal is to then add new parameter in the other write APIs to allow specifying specific bases that the user wants the new fragment data files to be written into

Yeah, I mean do we have any cases that use this api but not for writing data later?

[jackye1995]

Currently no. I think we could do things like create index or write deletion files also in other bases, the base path management API should only care about updating bases.

[wojiaodoubao]

I'm not sure If I understand it correctly, our goal is to store the dataset files (data files, index files, transaction files etc) across multiple buckets. I was thinking, could we use ObjectStore to achieve this? For example, we could refactor the current struct ObjectStore into trait ObjectStore, struct BaseObjectStore, and struct MultiObjectStore. Let MultiObjectStore handle the multi-destination issue.

MultiObjectStore can support various routing strategies to place files in different locations, such as Hash or Prefix(_transaction, _data etc). If we want to specify a particular write operation files must go to a specific bucket, we can include a store_tag during the write. The store_tag will be recorded in the file metadata, and the routing strategy will be responsible for parsing the store_tag and routing the file to the appropriate bucket.

The advantage is:

1. It has minimal intrusion into the lance table format code.
2. It can naturally adapt to all files, not only data files.
3. Because routing is encapsulated in the object store, we don't need to modify compaction & cleanup.

cc @jackye1995 @majin1102 @westonpace @jaystarshot