feat: support namespace as external manifest store

[jackye1995]

This PR is based on discussion in #5849

I have added 4 APIs in Lance Namespace spec:

• ListTableVersions (this already existed, but added a flag descending to allow return versions in descending order)
• CreateTableVersion
• DescribeTableVersion
• BatchDeleteTableVersions

With these APIs, we can fulfill the ExternalManfiestStore trait with a LanceNamespaceExternalManfiestStore by:

• get: call DescribeTableVersion
• get_latest_version: call ListTableVersions with limit=1, descending=false
• put: call CreateTableVersion (this is a new function that combines put_if_not_exist and put_if_exists)
• delete: call BatchDeleteTableVersions with range [0, -1)

We also added a basic implementation in DirectoryNamespace with:

• when CreateTableVersion, just put the staged manifest to the target version
• ListTableVersions, DescribeTableVersion, BatchDeleteTableVersions are just facade to storage level operations

In DirectoryNamespace, we added a flag table_version_tracking_enabled. When true, when user calls describe_table or declare_table to get table information, we return a new flag in response saying managed_versioning=true, indicating that the namespace should manage table version commit and resolution. Based on that information, we set the commit handler to be external manifest store commit handler, and the external manifest store to be LanceNamespaceExternalManfiestStore.

[github-actions]

PR Review

P0: Build will break - Invalid dependency path

Cargo.toml:69 changes lance-namespace-reqwest-client to a local path outside the repository (../lance-namespace/rust/lance-namespace-reqwest-client). This will fail for anyone without a sibling lance-namespace directory.

P1: Race condition in create_table_version

rust/lance-namespace-impls/src/dir.rs:693-721 has a TOCTOU race condition between checking if the version exists and copying the manifest. This defeats the put_if_not_exists semantics. Consider using copy_if_not_exists or similar atomic operation.

P1: put_if_exists contradicts implementation

rust/lance/src/io/commit/namespace_manifest.rs:114-133 - The put_if_exists method calls create_table_version, but create_table_version rejects existing versions. This needs a separate update API or reconciled semantics.

Summary

1. P0: Fix the Cargo.toml path dependency before merge
2. P1: The race condition undermines the core feature
3. P1: put_if_exists implementation is broken

[codecov]

Codecov Report

❌ Patch coverage is 82.69441% with 158 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-namespace-impls/src/dir.rs	89.07%	78 Missing and 8 partials ⚠️
rust/lance-namespace-impls/src/rest_adapter.rs	13.11%	53 Missing ⚠️
rust/lance/src/io/commit/namespace_manifest.rs	65.00%	7 Missing ⚠️
rust/lance-namespace/src/namespace.rs	0.00%	6 Missing ⚠️
rust/lance-namespace-impls/src/rest.rs	0.00%	4 Missing ⚠️
rust/lance/src/dataset.rs	91.66%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[jackye1995]

this is probably a good middle ground between external manifest store today vs doing a whole new manifest store. We will have a single generic put to put a manifest to the store that can be overridden. It gives more freedom compared to having to follow exactly the same flow with put_if_not_exist and then put_if_exists

[jackye1995]

looks like external manifest store code path in general has compatibility issue with windows, skip related tests for now and fix in a separated PR