Tracking: Distributed Indexes Search

[Xuanwo]

Lance is making distributed indexing a first-class capability by unifying all index families on a global segment model: a logical index is a set of physical segments, and build, commit, query, append, optimize, and GC all operate on segment sets.

Distributed indexing should not be a special path; it should be the natural consequence of a segment-native index system.

Workstreams

A. Segment Model and Transaction Semantics

• feat: clarify logical indices and physical index segments #6270
• fix(python): preserve index details in python index metadata #6279
• fix: preserve create index transaction semantics #6204, feat: add index segment commit API #6209
• refactor!: align distributed index build around segments #6313
• Enforce fragment_bitmap disjoint overlap at commit time
• Define ownership and cleanup for stale uncommitted index artifacts
• feat: support vector query pruning by index segments #6376

B. Vector: Multi-Segment as a Stable End State

• refactor: distributed vector segment build #6220, refactor!: remove staging from distributed vector indexing #6269
• test: tighten multi-segment vector index coverage and cleanup #6251
• feat: support non-shared centroid vector index builds #6296
• feat: support distributed IVF_RQ segment builds #6359
• Allow passing pre-trained rq metadata
• Make append and optimize segment-set-native #6398
• index reader for segmented index #6399

C. FTS: Segment-Native Distributed Indexing

• feat: support multiple index segments for FTS indices #6285
• feat: support segmented inverted index build and search #6305
• Define spill-to-segment mapping for workers that spill multiple times
• Compact / optimize as segment-set rebuild from source data
• Segment synopsis for query pruning
• Recompute WAND block-max upper bounds under global IDF

D. Lifecycle Management

• test: tighten multi-segment vector index coverage and cleanup #6251
• Crash / partial-failure recovery for in-flight distributed builds
• Cleanup sweep for stale uncommitted artifacts

E. Expansion to Other Index Families

• fix(python): preserve index details in python index metadata #6279 (scalar, single-source only)
• Define merge modes per family
• Decide which families need multi-segment query and distributed build support

References

• Lance Vector Index: Multi-Segment Final State #6189
• Design: Unify FTS into the Global Segment Model #6301