Skip to content

fix: panic if only one partition and split is triggered#5241

Merged
BubbleCal merged 6 commits intolance-format:mainfrom
BubbleCal:fix-spfresh
Nov 16, 2025
Merged

fix: panic if only one partition and split is triggered#5241
BubbleCal merged 6 commits intolance-format:mainfrom
BubbleCal:fix-spfresh

Conversation

@BubbleCal
Copy link
Copy Markdown
Contributor

@BubbleCal BubbleCal commented Nov 14, 2025

  • handle the case of single partition is split
  • this also fixes that single partition won't count the new data while checking split threshold
  • rewrite the tests to cover more cases

Signed-off-by: BubbleCal <bubble-cal@outlook.com>
Signed-off-by: BubbleCal <bubble-cal@outlook.com>
Signed-off-by: BubbleCal <bubble-cal@outlook.com>
Signed-off-by: BubbleCal <bubble-cal@outlook.com>
@github-actions github-actions Bot added the bug Something isn't working label Nov 14, 2025
}
}

pub struct SinglePartitionReader {
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't know how many rows in the reader, but SPFresh needs num_rows to determine whether to trigger a split job, so remove this

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would slow down indexing if num_partitions=1, but that means the dataset is small (otherwise users should indexing with more partitions), so it's fine

if min_dist <= d1 && min_dist <= d2 {
Ok(ReassignPartition::ReassignCandidate(
reassign_candidate_ids.value(min_dist_idx),
reassign_candidate_ids.value(min_dist_idx.unwrap()),
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here min_dist is not f32::INFINITY so min_dist_idx must be Some(...)

}
None => Ok(ReassignPartition::ReassignCandidate(
reassign_candidate_ids.value(min_dist_idx),
reassign_candidate_ids.value(min_dist_idx.unwrap()),
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a join job, which means there are at least 2 partitions, so min_dist_idx won't be None

Signed-off-by: BubbleCal <bubble-cal@outlook.com>
@codecov-commenter
Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.69725% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.24%. Comparing base (8c6732e) to head (111b367).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
rust/lance/src/index/vector/ivf/v2.rs 91.09% 17 Missing ⚠️
rust/lance/src/index/vector/ivf.rs 0.00% 3 Missing ⚠️
rust/lance-index/src/vector/hnsw/index.rs 0.00% 2 Missing ⚠️
rust/lance/src/index/vector/fixture_test.rs 0.00% 2 Missing ⚠️
rust/lance/src/index/vector/pq.rs 0.00% 2 Missing ⚠️
rust/lance/src/session/index_extension.rs 0.00% 2 Missing ⚠️
rust/lance/src/index/vector/builder.rs 93.75% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5241      +/-   ##
==========================================
+ Coverage   82.23%   82.24%   +0.01%     
==========================================
  Files         344      344              
  Lines      144755   144774      +19     
  Branches   144755   144774      +19     
==========================================
+ Hits       119033   119067      +34     
+ Misses      21800    21788      -12     
+ Partials     3922     3919       -3     
Flag Coverage Δ
unittests 82.24% <86.69%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@BubbleCal BubbleCal merged commit 417b9a8 into lance-format:main Nov 16, 2025
24 of 25 checks passed
jackye1995 pushed a commit to jackye1995/lance that referenced this pull request Jan 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants