feat(rust): support refresh frag bitmap for index after updating when… by yanghua · Pull Request #4589 · lance-format/lance

yanghua · 2025-08-28T06:57:16Z

… enable stable rowid

github-actions · 2025-08-28T14:27:12Z

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

codecov-commenter · 2025-08-30T09:05:08Z

Codecov Report

❌ Patch coverage is 96.94915% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.73%. Comparing base (419ac97) to head (df64a1c).
⚠️ Report is 13 commits behind head on main.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/transaction.rs	83.51%	13 Missing and 2 partials ⚠️
rust/lance/src/dataset/write/update.rs	98.84%	0 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4589      +/-   ##
==========================================
+ Coverage   80.61%   80.73%   +0.11%     
==========================================
  Files         320      321       +1     
  Lines      123816   125624    +1808     
  Branches   123816   125624    +1808     
==========================================
+ Hits        99814   101420    +1606     
- Misses      20437    20605     +168     
- Partials     3565     3599      +34

Flag	Coverage Δ
unittests	`80.73% <96.94%> (+0.11%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

yanghua · 2025-09-01T07:36:57Z

                fields_modified,
+                value_updated_fields: vec![],
                mem_wal_to_flush: self.params.mem_wal_to_flush,
+                update_mode: Some(VerticalFullSchema),


Actually, it belongs VerticalPartialSchema

@wjones127 It seems that even if it's a non-full-schema merge-insert, we still can not distinguish the updated fields(may contain the on field).

yanghua · 2025-09-03T23:46:12Z

@wjones127 When you have time, I hope you'll be able to answer. Very appreciated!

jbapple · 2025-09-08T16:11:31Z

@yanghua this is still marked as "Draft". Is that intentional?

yanghua · 2025-09-09T00:36:14Z

@yanghua this is still marked as "Draft". Is that intentional?

@jbapple Yeah, still need some questions, I am waiting for the answer from @wjones127

… enable stable rowid

yanghua · 2025-09-15T06:02:24Z

        mem_wal_to_merge: Option<MemWal>,
+        /// The fields that used to judge whether to preserve the new frag's id into
+        /// the frag bitmap of the specified indices.
+        fields_for_preserving_frag_bitmap: Vec<u32>,


To avoid confusion, name to this.

yanghua · 2025-09-15T07:48:03Z

@wjones127 This PR is ready for review, PTAL.

yanghua · 2025-09-17T16:13:54Z

@wjones127 Any input?

wjones127

This seems adequate for now. But I think before we say this is production ready, I'd recommend broader test coverage.

wjones127 · 2025-09-17T23:48:49Z

+        // the frag bitmap of the index for 'value' field should not been updated
+        let value_bitmap = updated_value_index.fragment_bitmap.as_ref().unwrap();
+        assert_eq!(value_bitmap.len(), 2);
+        assert!(value_bitmap.contains(0));
+        assert!(value_bitmap.contains(1));
+
+        let vec_bitmap = updated_vec_index.fragment_bitmap.as_ref().unwrap();
+        assert_eq!(vec_bitmap.len(), 2);
+        assert!(vec_bitmap.contains(0));
+        assert!(vec_bitmap.contains(1));


It seems like, as written, we'll never update these for upsert, right? Originally, I was thinking if stable row ids were enabled, then updated rows and new rows would be written to separate fragments so that we could do this.

There is a case for merge_insert where we only do updates. This could be just because there aren't any new rows. Or it could be we set WhenNotMatched::DoNothing. In either of these cases, do we never update the bitmap?

It seems like, as written, we'll never update these for upsert, right?

Yes, here(in this test) is because in full-schema mode, we did not figure out which columns are updated. So I set all fields to the fields_for_preserving_frag_bitmap, and it would cause the update for the frag bitmap not to happen.

Originally, I was thinking if stable row ids were enabled, then updated rows and new rows would be written to separate fragments so that we could do this.

IMO, it depends on whether we can distinguish which fields to update for each row in each fragment, right? It's hard to do this for RewriteRow mode. Correct me if I am wrong.

In either of these cases, do we never update the bitmap?

The current test cases are pure updates as you mentioned.

Currently, for merge insert, all the cases(update, upsert) would not update the frag bitmap. This is due to ReWriteRow mode, it's hard to split which fields would be updated. The detailed logic is here:

let index_covers_modified_field = index.fields.iter().any(|field_id| { value_updated_field_set.contains(&u32::try_from(*field_id).unwrap()) });

wjones127 · 2025-09-17T23:52:48Z

+    #[tokio::test]
+    async fn test_sub_schema_upsert_fragment_bitmap() {


Eventually, I'd like see a lot more tests cases for this. Unfortunately, right now the ratio of lines of code to useful tests cases is pretty high. I'd suggest thinking about how you can parameterize the tests to cover more useful situations. For example, you might use the same original test data, and very the merge insert parameters, input data, and index states.

Eventually, I'd like see a lot more tests cases for this.

Currently, the whole sub-schema merge insert would not be affected by this PR. It belongs to the RewriteColumn mode. It would update in-place. For pure update, the row id, fragment would not be updated, right?

yanghua · 2025-09-18T03:54:25Z

But I think before we say this is production ready, I'd recommend broader test coverage.

Absolutely, we can move faster. I would add more test cases and refactor them. Some logic may need to refactor, e.g. #4695 .

yanghua · 2025-09-18T06:01:40Z

Will merge it, if there are any issues, will fix them in the further PRs.

…4788) This PR enhances the Java bindings for Operation::Update as a follow-up to #4589. While #4589 introduced new fields to the Rust Operation::Update struct, the Java bindings were still missing support not only for those new fields but also for the existing `fields_modified field`, which is crucial for handling column modifications. Specifically, it adds support for the following fields: ```rust Update { // ... /// The fields that have been modified fields_modified: Vec<u32>, // 👇 New fields from #4589 /// The fields that used to judge whether to preserve the new frag's id into /// the frag bitmap of the specified indices. fields_for_preserving_frag_bitmap: Vec<u32>, /// The mode of update update_mode: Option<UpdateMode>, } ``` --------- Co-authored-by: Weiren <litaiwei.lwt@antgroup.com>

… enable stable rowid (lance-format#4589) … enable stable rowid

…ance-format#4788) This PR enhances the Java bindings for Operation::Update as a follow-up to lance-format#4589. While lance-format#4589 introduced new fields to the Rust Operation::Update struct, the Java bindings were still missing support not only for those new fields but also for the existing `fields_modified field`, which is crucial for handling column modifications. Specifically, it adds support for the following fields: ```rust Update { // ... /// The fields that have been modified fields_modified: Vec<u32>, // 👇 New fields from lance-format#4589 /// The fields that used to judge whether to preserve the new frag's id into /// the frag bitmap of the specified indices. fields_for_preserving_frag_bitmap: Vec<u32>, /// The mode of update update_mode: Option<UpdateMode>, } ``` --------- Co-authored-by: Weiren <litaiwei.lwt@antgroup.com>

github-actions Bot added the enhancement New feature or request label Aug 28, 2025

yanghua changed the title ~~feat(rust): support refresh frag bitmap for index after updating when…~~ [WIP] feat(rust): support refresh frag bitmap for index after updating when… Aug 28, 2025

yanghua force-pushed the stable-row-id-frag-bitmap branch from b07dd7c to 143db56 Compare August 30, 2025 13:59

github-actions Bot added python java labels Aug 31, 2025

yanghua force-pushed the stable-row-id-frag-bitmap branch from 4a9f1f4 to 1cfb738 Compare August 31, 2025 11:01

yanghua commented Sep 1, 2025

View reviewed changes

Comment thread rust/lance/src/dataset/transaction.rs

yanghua commented Sep 1, 2025

View reviewed changes

yanghua force-pushed the stable-row-id-frag-bitmap branch from 740a2c3 to 0d27710 Compare September 1, 2025 08:57

yanghua commented Sep 1, 2025

View reviewed changes

Comment thread rust/lance/src/index/vector/ivf/v2.rs Outdated

yanghua force-pushed the stable-row-id-frag-bitmap branch 4 times, most recently from 67b72d8 to a72cc51 Compare September 11, 2025 13:22

wjones127 reviewed Sep 12, 2025

View reviewed changes

Comment thread rust/lance/src/dataset/transaction.rs

Comment thread protos/transaction.proto

Comment thread rust/lance/src/index/vector/ivf/v2.rs Outdated

Comment thread rust/lance/src/dataset/transaction.rs Outdated

Comment thread rust/lance/src/index/vector/ivf/v2.rs Outdated

feat(rust): support refresh frag bitmap for index after updating when…

ce1cd1b

… enable stable rowid

yanghua force-pushed the stable-row-id-frag-bitmap branch 4 times, most recently from c57603d to e86c883 Compare September 14, 2025 12:35

yanghua marked this pull request as ready for review September 15, 2025 05:54

yanghua changed the title ~~[WIP] feat(rust): support refresh frag bitmap for index after updating when…~~ feat(rust): support refresh frag bitmap for index after updating when… Sep 15, 2025

yanghua commented Sep 15, 2025

View reviewed changes

address review comments

32c4b4e

yanghua force-pushed the stable-row-id-frag-bitmap branch from 3dfceda to 32c4b4e Compare September 15, 2025 06:17

wjones127 approved these changes Sep 17, 2025

View reviewed changes

address review comments

c81b9c4

fix format issue

df64a1c

yanghua merged commit a05d78d into lance-format:main Sep 18, 2025
26 checks passed

This was referenced Sep 19, 2025

feat: support fragment-level update columns #4715

Merged

feat(java): expose additional Operation::Update fields to bindings #4788

Merged

yanghua mentioned this pull request Sep 30, 2025

refactor: rename Update operation fields #4832

Closed

jackye1995 pushed a commit to jackye1995/lance that referenced this pull request Jan 21, 2026

feat(rust): support refresh frag bitmap for index after updating when…

162efdc

… enable stable rowid (lance-format#4589) … enable stable rowid

		#[tokio::test]
		async fn test_sub_schema_upsert_fragment_bitmap() {

Conversation

yanghua commented Aug 28, 2025

Uh oh!

github-actions Bot commented Aug 28, 2025

Uh oh!

codecov-commenter commented Aug 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

yanghua Sep 1, 2025

Choose a reason for hiding this comment

Uh oh!

yanghua Sep 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yanghua commented Sep 3, 2025

Uh oh!

jbapple commented Sep 8, 2025

Uh oh!

yanghua commented Sep 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yanghua Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

yanghua commented Sep 15, 2025

Uh oh!

yanghua commented Sep 17, 2025

Uh oh!

wjones127 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wjones127 Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

yanghua Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wjones127 Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

yanghua Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

yanghua commented Sep 18, 2025

Uh oh!

yanghua commented Sep 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-commenter commented Aug 30, 2025 •

edited

Loading

yanghua Sep 18, 2025 •

edited

Loading