Concurrent commit overwrites + OSS If-None-Match write failures (Lance ≥ 0.24) – any mature commit_lock solution?

[Pmathsun]

Hi,I'm hitting two problems:

• Concurrent commits from multiple processes can overwrite each other.

• With Lance ≥ 0.24.0, writing to Alibaba OSS fails because the backend doesn’t honor the If-None-Match header.

My idea is to plug in a commit_lock implementation to guard the commit step, but before I build my own, is there already a proven solution out there? Any pointers would be appreciated.

[Xuanwo]

One possible approach is to use OpenDAL's native OSS support, which implements the if-none-match feature based on OSS's x-oss-forbid-overwrite: https://opendal.apache.org/docs/rust/opendal/services/struct.Oss.html

However, OpenDAL is not yet integrated with Lance, so we need further discussion on how to proceed.

[jiaoew1991]

I encountered this issue in other repos as well, and it was eventually solved using opendal. 😁slatedb/slatedb#636

[westonpace]

I think OpenDAL integration would be awesome.

My idea is to plug in a commit_lock implementation to guard the commit step, but before I build my own, is there already a proven solution out there?

No, I'm not aware of any standard implementation. The most robust alternative that we have used was dynamodb. We needed this because S3 itself didn't support If-None-Match when we started. However, we implemented the dynamodb integration at the rust level and not python. In Rust the trait is ExternalManifestStore.

[Pmathsun]

I think OpenDAL integration would be awesome.我认为 OpenDAL 集成会很棒。

My idea is to plug in a commit_lock implementation to guard the commit step, but before I build my own, is there already a proven solution out there?我的想法是插入一个 commit_lock 实现来保护提交步骤，但在我构建自己的之前，是否已经有一个经过验证的解决方案？

No, I'm not aware of any standard implementation. The most robust alternative that we have used was dynamodb. We needed this because S3 itself didn't support If-None-Match when we started. However, we implemented the dynamodb integration at the rust level and not python. In Rust the trait is ExternalManifestStore.不，我不知道任何标准实现。我们使用过的最强大的替代方案是 DynamoDB。我们需要这个，因为 S3 本身在我们开始时不支持 If-None-Match。但是，我们在 rust 级别而不是 python 级别实施了 dynamodb 集成 。在 Rust 中，trait 是 ExternalManifestStore。

Got it. Are there any plans to add S3 support in Python in the near future?

[Xuanwo]

Hi @Pmathsun, our native support for oss should address your issue: #4247, let me know if there are anything not work.