fix!: check metric compatibility before using vector index

[wjones127]

BREAKING CHANGE: Query.metric_type is now Option<DistanceType> instead of DistanceType and the default is None. This means that it will default to whatever the vector index distance type is, or L2 (the old default if there is no index built). The other breaking change is that if you explicitly pass a distance type but the index was built on a different distance, it will now fall back to flat search instead of overriding the distance type.

Summary

• Fix vector search using ANN index regardless of query's metric type
• Query.metric_type is now Option<DistanceType> (None = use index default)
• If user specifies a metric that doesn't match the index, fall back to flat search
• Explain plan now shows the metric being used

Test plan

• Added regression test test_knn_metric_mismatch_falls_back_to_flat_search
• Added test test_knn_no_metric_uses_index_metric
• Existing tests pass with updated explain plan expectations

Fixes #5608

🤖 Generated with Claude Code

[github-actions]

Code Review

Summary: This PR fixes a critical bug (#5608) where vector search incorrectly used the ANN index regardless of the query's metric type, returning wrong distances. The fix changes Query.metric_type from DistanceType to Option<DistanceType>, enabling metric compatibility checking before index selection.

No Major Issues Found

The implementation is correct and well-tested. The logic properly:

1. Falls back to flat search when user-specified metric doesn't match the index
2. Uses the index's metric when no preference is specified
3. Resolves to a default metric for flat search when needed

Minor Observations (Non-blocking)

1. Double index open in scanner.rs (lines ~3037-3066)

The index is opened to check metric compatibility, but then _idx is discarded and the index is used again only via matching_index. This is fine since the opened index object isn't needed after getting the metric type, but you might consider documenting why the pattern is structured this way if it's intentional for future extensibility.

2. Java API backward compatibility

The change from required distanceType (defaulting to L2) to Optional<DistanceType> is a semantic change. Previously:

new Query.Builder().setKey(...).build()  // Used L2

Now uses index default or data-type default. This is the intended behavior per the fix, but downstream Java users relying on implicit L2 may see different results if their index uses a different metric. This seems acceptable for a bug fix, but worth noting in release notes.

Tests

Good test coverage with test_knn_metric_mismatch_falls_back_to_flat_search and test_knn_no_metric_uses_index_metric.

LGTM

[codecov]

Codecov Report

❌ Patch coverage is 96.26168% with 4 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/scanner.rs	97.91%	0 Missing and 2 partials ⚠️
rust/lance/src/io/exec/knn.rs	77.77%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[BubbleCal]

Thanks for this work!

[rpgreen]

While this is a bugfix, it could be a surprising behavior change. Should it be flagged as a breaking change?