feat: add adapter for REST namespace with manifest namespace backend#4984
Merged
jackye1995 merged 19 commits intolance-format:mainfrom Nov 14, 2025
Merged
feat: add adapter for REST namespace with manifest namespace backend#4984jackye1995 merged 19 commits intolance-format:mainfrom
jackye1995 merged 19 commits intolance-format:mainfrom
Conversation
jackye1995
added a commit
that referenced
this pull request
Oct 23, 2025
…lder (#5045) I ended up doing these in #4984 and #4905 so I decided to pull it out and get it cleaned up first. This PR moves the directory namespace from using OpenDAL directly to using Lance ObjectStore. This avoids the inconsistency between the dir namespace and the underlying lance table storage configurations. User can still use OpenDAL, and if we fully migrate Lance to OpenDAL it will be applied to both layers at the same time as well. The PR also improves the builder of the namespaces with builder style and allow supplying a Lance session. Since we have not published a stable version yet, we do not care about backwards compatibility. This PR also ensures the lance-namespace-impls features are consistent with lance-io features. Related to #5042
jackye1995
force-pushed
the
rest-adapter
branch
from
f6d85cd to
6f80791
Compare
jackye1995
force-pushed
the
rest-adapter
branch
2 times, most recently
from
e8a1fbd to
5f1717c
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #4984 +/- ##
==========================================
+ Coverage 82.02% 82.21% +0.19%
==========================================
Files 342 344 +2
Lines 141495 144687 +3192
Branches 141495 144687 +3192
==========================================
+ Hits 116059 118959 +2900
- Misses 21595 21807 +212
- Partials 3841 3921 +80
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
jackye1995
force-pushed
the
rest-adapter
branch
from
5f1717c to
fa2c3bb
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
jackye1995
changed the title
jackye1995
added a commit
to lance-format/lance-namespace
that referenced
this pull request
Nov 14, 2025
…ls (#258) These are now available with lance-format/lance#4984
Xuanwo
approved these changes
Nov 14, 2025
jackye1995
added a commit
to lancedb/lancedb
that referenced
this pull request
Nov 17, 2025
Based on lance-format/lance#4984 1. Bump to 1.0.0-beta.2 2. Use DirectoryNamespace in lance to perform all testing in python and rust for much better coverage 3. Refactor `ListingDatabase` to be able to accept location and namespace. This is because we have to leverage listing database (local lancedb connection) for using namespace, namespace only resolves the location and storage options but we don't want to bind all the way to rust since user will plug-in namespace from python side. And thus `ListingDatabase` needs to be able to accept location and namespace that are created from namespace connection. 4. For credentials vending, we also pass storage options provider all the way to rust layer, and the rust layer calls back to the python function to fetch next storage option. This is exactly the same thing we did in pylance.
jackye1995
added a commit
to lance-format/lance-namespace
that referenced
this pull request
Nov 26, 2025
Based on implementation in lance-format/lance#4984
jackye1995
added a commit
to jackye1995/lance
that referenced
this pull request
Jan 21, 2026
…lder (lance-format#5045) I ended up doing these in lance-format#4984 and lance-format#4905 so I decided to pull it out and get it cleaned up first. This PR moves the directory namespace from using OpenDAL directly to using Lance ObjectStore. This avoids the inconsistency between the dir namespace and the underlying lance table storage configurations. User can still use OpenDAL, and if we fully migrate Lance to OpenDAL it will be applied to both layers at the same time as well. The PR also improves the builder of the namespaces with builder style and allow supplying a Lance session. Since we have not published a stable version yet, we do not care about backwards compatibility. This PR also ensures the lance-namespace-impls features are consistent with lance-io features. Related to lance-format#5042
jackye1995
added a commit
to jackye1995/lance
that referenced
this pull request
Jan 21, 2026
…ance-format#4984) This PR introduces an adapter (using Axum) that provides a REST server which delegates requests to be fulfilled by another Lance namespace. This is helpful for easy setup of a REST namespace server to perform testing. In addition, I added a namespace implementation backed by a Lance table, basically the same thing as lancedb/lancedb#2708, also have DirectoryNamespace able to use it in a compatibility mode. I moved it here as a namespace implementation, because (1) this allows us to fully test tables in a namespace with storage only requirement, and at the same time, (2) comparing to offer this in lancedb directly, we know we want to consolidate the local listing connection and remote connection as just 2 types of namespace connections, and this gives user an incentive to move to the namespace-based connection in lancedb for the local one Also add corresponding python binding for DirectoryNamespace and RestNamespace, deprecate the current ones in lance-namespace python package
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR introduces an adapter (using Axum) that provides a REST server which delegates requests to be fulfilled by another Lance namespace. This is helpful for easy setup of a REST namespace server to perform testing.
In addition, I added a namespace implementation backed by a Lance table, basically the same thing as lancedb/lancedb#2708, also have DirectoryNamespace able to use it in a compatibility mode.
I moved it here as a namespace implementation, because (1) this allows us to fully test tables in a namespace with storage only requirement, and at the same time, (2) comparing to offer this in lancedb directly, we know we want to consolidate the local listing connection and remote connection as just 2 types of namespace connections, and this gives user an incentive to move to the namespace-based connection in lancedb for the local one
Also add corresponding python binding for DirectoryNamespace and RestNamespace, deprecate the current ones in lance-namespace python package