feat: add index segment commit API by Xuanwo · Pull Request #6209 · lance-format/lance

Xuanwo · 2026-03-17T02:59:20Z

This adds a first-class IndexSegment type and a commit_existing_index_segments API so a logical index can be committed from multiple physical segments. It is intended as a building block for the multi-segment vector index work without changing the high-level create_index flow.

Part of #6180

github-actions · 2026-03-17T03:00:18Z

PR Review: feat: add index segment commit API

Overall this is a clean, well-tested PR. A few items worth discussing:

P1: Truncation casting `u64` fragment id → `u32`

Fragment.id is u64, but iter_fragment_ids() casts via f.id as u32, which silently truncates if a fragment id ever exceeds u32::MAX. This pre-existing pattern is used elsewhere (scanner, get_fragments), but codifying it in a new public API method is a good time to add a safety check or at least document the invariant. Consider u32::try_from(f.id).expect("fragment id exceeds u32") or a comment explaining why truncation is safe.

P1: `IndexSegment` duplicates every field of `IndexMetadata` (minus `name`/`fields`/`dataset_version`)

IndexSegment has the exact same field types and names as the corresponding IndexMetadata fields. This creates a maintenance risk — any future field added to IndexMetadata (e.g., a new optional attribute) must be manually mirrored. Consider either:

Making IndexSegment contain an IndexMetadata internally and having the commit API fill in the shared fields, or
Adding a From<IndexSegment> / builder that takes the shared fields as parameters

This is a design suggestion, not a blocker — but worth thinking about before the type becomes part of the public API surface.

Minor

The chrono dependency added to lance-index (Cargo.toml) appears unused — IndexSegment lives in lance-table, and the traits file doesn't use chrono directly. Verify this is needed.

Tests look thorough — good coverage of the happy path, duplicate-segment rejection, empty-segments rejection, and backward-compat wrapper.

codecov · 2026-03-17T04:37:26Z

Codecov Report

❌ Patch coverage is 94.64286% with 9 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-index/src/types.rs	74.28%	9 Missing ⚠️

📢 Thoughts on this report? Let us know!

westonpace

Some suggestions but looks good overall

westonpace · 2026-03-18T12:35:39Z

+                    index_details,
+                    index_version,
+                    created_at: Some(chrono::Utc::now()),
+                    base_id: None,


Is there no way to specify a base id? Not that important for now but I think we'll eventually need it.

IndexSegment exposed this API but no upper APIs to use it, we need to refactor the upper APIs first. Yes, we can treat this as a follow up.

…commit-api # Conflicts: # rust/lance/src/index.rs

The method was renamed in lance-format#6209 but the test call site in v2.rs was not updated during the merge. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

This refactors distributed vector indexing into a staged segment-build pipeline and exposes the public APIs needed to integrate an external distributed build workflow with Lance. It defines the storage-level model for partial shards, staged planning, built segments, and segment commit, and documents the current distributed indexing flow. This PR builds on the segment commit API from #6209. The main changes are organized into five commits: - [test: cover distributed vector segment build](a86274da2) - [refactor: internalize distributed vector segment build](1e5f0e15b) - [feat: add public vector segment builder API](691cecb9a) - [feat: add Python vector segment builder API](a07ef6144) - [docs: document distributed vector segment build](0a3f230d7) Follow-up fixes: - [fix: expose python create_index_uncommitted](c1d3b1666) - [fix: format python bindings](b86f91c4d) - [fix: format python segment builder bindings](bfc9e63a0) Please review accordingly. --- Guide: https://github.com/lance-format/lance/blob/xuanwo/vector-staging-merge-internal/docs/src/guide/distributed_indexing.md

This adds a first-class `IndexSegment` type and a `commit_existing_index_segments` API so a logical index can be committed from multiple physical segments. It is intended as a building block for the multi-segment vector index work without changing the high-level `create_index` flow. Part of lance-format#6180

The method was renamed in lance-format#6209 but the test call site in v2.rs was not updated during the merge. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

This refactors distributed vector indexing into a staged segment-build pipeline and exposes the public APIs needed to integrate an external distributed build workflow with Lance. It defines the storage-level model for partial shards, staged planning, built segments, and segment commit, and documents the current distributed indexing flow. This PR builds on the segment commit API from lance-format#6209. The main changes are organized into five commits: - [test: cover distributed vector segment build](lance-format@a86274da2) - [refactor: internalize distributed vector segment build](lance-format@1e5f0e15b) - [feat: add public vector segment builder API](lance-format@691cecb9a) - [feat: add Python vector segment builder API](lance-format@a07ef6144) - [docs: document distributed vector segment build](lance-format@0a3f230d7) Follow-up fixes: - [fix: expose python create_index_uncommitted](lance-format@c1d3b1666) - [fix: format python bindings](lance-format@b86f91c4d) - [fix: format python segment builder bindings](lance-format@bfc9e63a0) Please review accordingly. --- Guide: https://github.com/lance-format/lance/blob/xuanwo/vector-staging-merge-internal/docs/src/guide/distributed_indexing.md

feat: add index segment commit API

2524ec8

github-actions Bot added the enhancement New feature or request label Mar 17, 2026

Xuanwo added 4 commits March 17, 2026 11:14

refactor: reuse dataset fragment bitmap

260ab08

refactor: narrow index segment API

1ee8aad

Merge branch 'main' into xuanwo/index-segment-commit-api

9466b15

build: update binding lockfiles

a66d616

github-actions Bot added python java labels Mar 17, 2026

Merge branch 'main' into xuanwo/index-segment-commit-api

ea9b85c

Xuanwo added 2 commits March 17, 2026 12:46

refactor: encapsulate index segment construction

4b330de

style: fix python indices formatting

575670b

Xuanwo mentioned this pull request Mar 17, 2026

Tracking Issues for Vector Index Segments #6180

Open

2 tasks

Merge branch 'main' into xuanwo/index-segment-commit-api

98c707e

Xuanwo mentioned this pull request Mar 18, 2026

refactor: distributed vector segment build #6220

Merged

westonpace approved these changes Mar 18, 2026

View reviewed changes

refactor: tighten index segment follow-ups

ecd23f0

Xuanwo mentioned this pull request Mar 18, 2026

Move DatasetIndexExt out of lance-index #6221

Closed

Xuanwo added 4 commits March 19, 2026 00:36

refactor: require complete index segment metadata

a1a805f

Merge remote-tracking branch 'origin/main' into xuanwo/index-segment-…

aa6c77e

…commit-api # Conflicts: # rust/lance/src/index.rs

refactor: require index segment commit support

4851fb6

fix: remove unnecessary index version casts

975c1f5

Xuanwo merged commit c9e5b1e into main Mar 18, 2026
28 checks passed

Xuanwo deleted the xuanwo/index-segment-commit-api branch March 18, 2026 17:22

Xuanwo added the breaking-change label Mar 21, 2026

Xuanwo mentioned this pull request Mar 27, 2026

Tracking: Distributed Indexes Search #6309

Open

26 tasks

jiaoew1991 mentioned this pull request Apr 25, 2026

feat: support distributed IVF vector index builds lance-format/lance-spark#479

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add index segment commit API#6209

feat: add index segment commit API#6209
Xuanwo merged 14 commits intomainfrom
xuanwo/index-segment-commit-api

Xuanwo commented Mar 17, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Mar 17, 2026

Uh oh!

codecov Bot commented Mar 17, 2026 •

edited

Loading

Uh oh!

westonpace left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

westonpace Mar 18, 2026

Uh oh!

Xuanwo Mar 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Xuanwo commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Mar 17, 2026

PR Review: feat: add index segment commit API

P1: Truncation casting u64 fragment id → u32

P1: IndexSegment duplicates every field of IndexMetadata (minus name/fields/dataset_version)

Minor

Uh oh!

codecov Bot commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

westonpace Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

Xuanwo Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Xuanwo commented Mar 17, 2026 •

edited

Loading

P1: Truncation casting `u64` fragment id → `u32`

P1: `IndexSegment` duplicates every field of `IndexMetadata` (minus `name`/`fields`/`dataset_version`)

codecov Bot commented Mar 17, 2026 •

edited

Loading