refactor: distributed vector segment build

[Xuanwo]

This refactors distributed vector indexing into a staged segment-build pipeline and exposes the public APIs needed to integrate an external distributed build workflow with Lance. It defines the storage-level model for partial shards, staged planning, built segments, and segment commit, and documents the current distributed indexing flow.

This PR builds on the segment commit API from #6209. The main changes are organized into five commits:

• test: cover distributed vector segment build
• refactor: internalize distributed vector segment build
• feat: add public vector segment builder API
• feat: add Python vector segment builder API
• docs: document distributed vector segment build

Follow-up fixes:

• fix: expose python create_index_uncommitted
• fix: format python bindings
• fix: format python segment builder bindings

Please review accordingly.

---

Guide: https://github.com/lance-format/lance/blob/xuanwo/vector-staging-merge-internal/docs/src/guide/distributed_indexing.md

[github-actions]

PR Review: refactor: internalize vector staging merge flow

Overall: Well-structured refactor that cleanly separates planning from execution for distributed vector index finalize. Good test coverage across IVF subtypes and error cases. A few items to consider:

P1: load_fragment_bitmap_from_storage scans all partitions and all row IDs

load_fragment_bitmap_from_storage (ivf.rs ~line 2598) iterates every partition and every row ID to reconstruct the fragment bitmap. For large indices with millions of vectors, this could be very expensive during plan_staging_segments (which calls load_partial_vector_segment per shard) and also during load_vector_segment in commit_existing_index. This happens at planning time, not just at merge time.

Consider whether the fragment bitmap could be stored as a sidecar in the partial shard directory during build, or at minimum document the cost tradeoff and ensure callers are aware. For the legacy compat path (commit_existing_index → load_vector_segment) this could be a regression if the index is large.

P1: reset_final_segment_dir deletes before writing — no atomicity

merge_staging_segment_to_dir calls reset_final_segment_dir (which does remove_dir_all) on the final_dir before writing new content. If the process crashes between delete and write completion, the segment is lost. The legacy same-dir path is especially fragile: it deletes the temp dir, copies files to final_dir, then deletes the partial shards — multiple non-atomic steps.

For object stores (S3/GCS) this is somewhat inherent, but for local filesystem consider whether an atomic rename could be used instead, or at least note the non-atomic window in comments.

Minor observations

• copy_partial_segment_contents does serial object_store.copy() per file. For cloud stores with many files per shard, parallel copies (e.g., futures::stream::iter(...).buffered(N)) would be faster.
• plan_staging_segments processes shards sequentially in a for loop. Since each load_partial_vector_segment does I/O, these could be loaded concurrently with try_join_all or similar.
• The chrono dependency added to lance-index is only used for IndexSegment's created_at field in the commit path (in lance/src/index.rs), not in lance-index itself. It doesn't appear to be used in types.rs. Is this dependency needed in lance-index?

Tests

Good coverage: multiple IVF subtypes (flat/pq/sq), HNSW variants, error cases for mismatched subtypes/centroids/overlapping fragments, duplicate segments, empty segments, and the legacy commit_existing_index wrapper. The parametrized rstest approach is clean.

[Xuanwo]

CI is not running because our base is not main. I will change that once #6209 merged.

[codecov]

Codecov Report

❌ Patch coverage is 91.45129% with 86 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/index/vector/ivf.rs	80.96%	34 Missing and 21 partials ⚠️
rust/lance/src/dataset.rs	73.91%	7 Missing and 5 partials ⚠️
rust/lance/src/index/create.rs	96.66%	9 Missing and 1 partial ⚠️
rust/lance/src/index/vector.rs	82.14%	5 Missing ⚠️
...lance-index/src/vector/distributed/index_merger.rs	92.50%	3 Missing ⚠️
rust/lance/src/index/vector/ivf/v2.rs	99.62%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[westonpace]

Still trying to get up to speed I think. What's the difference between merging staging files into a segment and combining multiple smaller segments into a larger segment?

[Xuanwo]

Thank you @westonpace for the review! I have revisited all the concepts, aligned their naming and comments, and compiled them into a document at https://github.com/lance-format/lance/blob/xuanwo/vector-staging-merge-internal/docs/src/guide/distributed_indexing.md. I hope this will be helpful for the review.

[westonpace]

This guide is great, and super helpful. I have a few more questions but we can potentially address these in a follow-up.

How does a worker decide how many shards to create?
Why does each worker create shards and not index segments? For example, let's say I have 5 workers and each worker created 10 shards so I have 50 shards.

Are those shards complete indexes? Why not just make them segments?

Then there is no need for a staging dir. The create_index_segment_builder call would take in a list of segment UUIDs (instead of a staging dir). I don't think you'd need the call with_partial_shards (since we provide these to create_index_segment_builder. Everything else would work as planned.

[westonpace]

Can you clearly document what the breaking changes are in the PR description?

Overall this seems to be a good extension to the previous single segment mechanism. It is flexible and the API makes sense.

It might be nice to see a full end-to-end example (perhaps in distributed_indexing.md) at some point.

[westonpace]

Is this a breaking change?

[westonpace]

It's hard to tell how this changed from the existing API. Were there any breaking changes?

[Xuanwo]

Can you clearly document what the breaking changes are in the PR description?

I think we should mark #6209 as a breaking change. Therefore, this PR does not necessarily need to be a breaking change now. I will update.

It might be nice to see a full end-to-end example (perhaps in distributed_indexing.md) at some point.

Yes! I'm thinking about this too, will work on a follow-up.