fix: concurrent read and write to directory namespace

[jackye1995]

Summary

• Add Throttled error type (code 21) to lance-namespace for rate limiting and concurrent operation throttling
• Add commit_retries config to DirectoryNamespace builder (default 20) to control inner commit retry count.
• For MergeInserts run in DirectoryNamespace, make sure those set outer to 0.
• Ensure object_id is set as the primary key in the __manifest table to ensure we enable primary-key-based dedupe.
• Add commit_retries() method to MergeInsertBuilder for controlling inner manifest write retry
• Properly map error types: CommitConflict → Throttled (safe to retry), TooMuchWriteContention → ConcurrentModification (semantic conflict)
• Add comprehensive concurrent create/drop tests for Python, Java, and Rust with S3 backend

Notes:

1. Also tested with lance-trino to make sure it solves the problems in Trino for concurrent reads and writes
2. I added tests in both java and python to make sure the binding does not introduce additional issue for concurrent access.

🤖 Generated with Claude Code

[github-actions]

PR Review: feat(namespace): add Throttled error and configurable commit retries

Overall, this PR adds important concurrency handling for namespace operations. The test coverage is comprehensive. A few items to consider:

P1: Semantic concern with error mapping

The mapping of CommitConflict → Throttled (lines in manifest.rs:831-841) may be semantically misleading:

if error_msg.contains("CommitConflict")
    || error_msg.contains("Failed to commit the transaction after")
{
    NamespaceError::Throttled { ... }
}

"Throttled" typically implies rate limiting by the service, while CommitConflict indicates version conflicts from concurrent writes. Clients might expect different retry strategies for each:

• Throttled → exponential backoff respecting service limits
• CommitConflict → immediate retry with fresh state

Consider using a more specific error variant like CommitRetryExhausted or documenting clearly that Throttled is used for all retriable contention scenarios.

Minor: Unnecessary clone in delete_from_manifest

self.manifest_dataset
    .set_latest(Arc::try_unwrap(new_dataset).unwrap_or_else(|arc| (*arc).clone()))
    .await;

Since new_dataset is an Arc<Dataset> returned from DeleteBuilder::execute(), try_unwrap will likely always fail (reference count > 1), resulting in an unnecessary clone. Consider just using Arc::unwrap_or_clone (Rust 1.76+) or accepting the Arc directly if the API allows.

Note: High retry counts in tests

Tests use commit_retries: "1000" which is fine for testing, but ensure the default of 20 is sufficient for production S3 workloads where latency can vary significantly.

---

The concurrent tests are well-structured and mirror the same patterns across Rust, Python, and Java. The debug logging additions will be helpful for troubleshooting production issues.

[jackye1995]

there are in general some problems in windows path, that I will fix fully in a separated PR

[wjones127]

I think some of the errors might be mixed up.

I like the tests though.

[codecov]

Codecov Report

❌ Patch coverage is 54.36508% with 115 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-namespace-impls/src/dir/manifest.rs	56.33%	88 Missing and 12 partials ⚠️
rust/lance/src/dataset/write/merge_insert.rs	12.50%	5 Missing and 2 partials ⚠️
rust/lance-namespace-impls/src/dir.rs	50.00%	4 Missing and 1 partial ⚠️
rust/lance-namespace/src/error.rs	40.00%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!