fix(index): preserve stable row-id entries during scalar index optimize

[acking-you]

Preserve stable row-id entries during scalar index optimize

Fixes #6116

Summary

This PR fixes a bug where optimize_indices() could drop valid BTree index entries when the dataset used stable row IDs.

I hit this while building the music module of StaticFlow, my personal project built on top of Lance/LanceDB. The songs dataset uses:

• enable_stable_row_ids = true
• a BTree scalar index on id

After running:

1. compact_files()
2. optimize_indices()

full scans still returned the expected rows, but indexed equality lookups such as id = 'song-42' returned no rows.

Root Cause

The old optimize path filtered old BTree rows with logic equivalent to:

valid_fragments.contains((row_id >> 32) as u32)

That is correct for address-style row IDs:

row_id = (fragment_id << 32) | row_offset

But it is incorrect for stable row IDs, because stable row IDs are opaque logical IDs and do not encode fragment ownership in their upper bits.

As a result, valid old index rows could be removed during optimize even though the underlying rows were still present after compaction.

What This PR Changes

• adds an explicit old-data filter mode to scalar index update
• keeps fragment-based filtering for address-style row IDs
• builds an exact retained row-ID set for stable-row-ID datasets from persisted row-id sequences
• filters old BTree rows by exact row-ID membership for the stable-row-ID case
• adds regression coverage for both the BTree update path and the end-to-end compaction plus optimize flow

Implementation Notes

The key change is to stop assuming that every row_id can be interpreted as a row address.

For stable-row-ID datasets, the optimize path now:

1. computes the retained old fragments
2. loads their row-ID sequences
3. builds one exact retained row-ID set
4. keeps only old index rows whose row IDs are still valid

This preserves the existing fast path for address-style row IDs and only uses exact row-ID filtering when the dataset actually uses stable row IDs.

Additional Context

I also wrote a longer deep dive covering the bug, the stable-row-ID model, and the full repair process:

• https://ackingliu.top/posts/lance-stable-row-id-deep-dive

Final Note

This PR document and parts of the implementation work were prepared with assistance from Codex GPT-5.4. The final patch has been reviewed by me personally, and it has been running normally in my production StaticFlow environment for one week.

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[github-actions]

PR Review

Summary: This PR fixes a real bug where optimize_indices() drops valid BTree entries for datasets using stable row IDs, because the existing filter incorrectly interprets opaque stable row IDs as address-encoded (fragment_id in upper 32 bits). The fix is correct and well-structured.

Design

The approach of introducing OldIndexDataFilter as an enum with Fragments vs RowIds variants is clean. It preserves the fast path for address-style row IDs and only activates the exact-membership path when needed. The trait signature change across all scalar index types is mechanical and correct — only BTree actually uses the filter.

One concern: deletion vectors not accounted for

In build_stable_row_id_filter, the retained row-ID set is built from the full row-id sequences of retained fragments, without subtracting rows that have been soft-deleted (deletion vectors). This means the allow-list may include IDs for deleted rows.

However, this matches the existing behavior of the fragment-based filter for address-style row IDs — filter_row_ids_by_fragments also doesn't exclude soft-deleted rows within valid fragments. So this is not a regression, but it may be worth a brief comment noting this parity.

Tests

Good coverage: both a unit test (test_update_with_exact_row_id_filter) exercising the BTree filter path directly, and an end-to-end integration test (test_optimize_btree_keeps_rows_with_stable_row_ids_after_compaction) covering the full write → index → compact → optimize → query flow.

Minor nit

In build_stable_row_id_filter, the pattern of building individual RowAddrTreeMap per fragment then calling union_all allocates N intermediate maps. You could avoid this by inserting directly into a single RowAddrTreeMap in a fold:

let mut result = RowAddrTreeMap::new();
for (_, seq) in row_id_sequences {
    let map = RowAddrTreeMap::from(seq.as_ref());
    result = RowAddrTreeMap::union(&result, &map);
}

Not blocking — the current approach is clear and correct, just slightly more allocations than necessary.

Overall this looks good. The bug analysis is solid, the fix is targeted, and the tests are appropriate.

[acking-you]

@Xuanwo could you please help review this PR when you have time?

This bug is currently blocking my normal production use of LanceDB. In my case it causes indexed equality lookups to fail after compact_files() + optimize_indices() on datasets with stable
row IDs enabled, even though the underlying rows are still present.

I also wrote up the full analysis and fix here:
https://ackingliu.top/posts/lance-stable-row-id-deep-dive

Thanks a lot.

[wjones127]

This fix looks correct to me, but I'd like the implementation to be refactored a bit so that each index index implementation isn't exposed to the details of old vs new style ids.

[wjones127]

It would be nice if OldIndexDataFilter encapsulated this distinction within.

Maybe something like:

impl OldIndexDataFilter {
    pub fn filter_row_ids(&self, row_ids: &UInt64Array) -> BooleanArray {
        match self {
            Self::Fragments(frags) => ...,
            Self::RowIds(row_ids) => ...,
    }
}

[acking-you]

Thanks, this is a good suggestion

[codecov]

Codecov Report

❌ Patch coverage is 97.22222% with 5 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/index/append.rs	96.90%	1 Missing and 2 partials ⚠️
rust/lance-index/src/scalar/btree.rs	97.22%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!