MB-69655: Fix vector normalization to handle multi-vectors correctly #2260
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CascadingRadium
force-pushed
the
cosineFix
branch
from
2f2e259 to
4e3891f
Compare
CascadingRadium
requested review from
Likith101,
Thejas-bhat,
abhinavdangeti,
capemox,
Copilot and
maneuvertomars
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a critical bug in multi-vector normalization for cosine similarity searches. Previously, when indexing multi-vector fields (e.g., [[3,0,0], [0,4,0]]), the normalization was incorrectly applied to the entire flattened array rather than normalizing each sub-vector independently, resulting in incorrect similarity scores.
Key changes:
- Added
NormalizeMultiVector(vec, dims)function that normalizes each sub-vector separately for multi-vector fields - Updated normalization calls in
processVectorandprocessVectorBase64to use the new function - Refactored
NormalizeVectorto useslices.Cloneinstead of manual copying - Added comprehensive unit tests for multi-vector normalization with various edge cases
- Added end-to-end integration test verifying correct cosine similarity scores for multi-vector fields
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
mapping/mapping_vectors.go |
Implements NormalizeMultiVector to normalize each sub-vector independently; updates normalization calls to use new function; refactors NormalizeVector to use slices.Clone |
mapping/mapping_vectors_test.go |
Adds comprehensive unit tests for NormalizeMultiVector covering single vectors, multi-vectors, edge cases (empty, zero/negative dims), and helper functions for magnitude computation and float comparison |
search_knn_test.go |
Adds end-to-end integration test TestMultiVectorCosineNormalization verifying correct similarity scores (1.0 for exact matches) on both single-vector and multi-vector fields with cosine similarity |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Likith101
previously approved these changes
Dec 4, 2025
abhinavdangeti
dismissed
Likith101’s stale review
The base branch was changed.
Member
|
Changed the base branch to |
Member
Author
|
hey @abhinavdangeti, i had set the branch to |
Member
|
Ah ok, let's merge this only after that goes in then. |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
abhinavdangeti
approved these changes
Dec 5, 2025
Member
abhinavdangeti
left a comment
abhinavdangeti
left a comment
There was a problem hiding this comment.
Looks good 👍🏼 .
@CascadingRadium lets merge this one only after the knnDup branch is merged into master and the base branch here is changed to master - so we produce 2 separate commits for these two separate issues.
capemox
approved these changes
Dec 8, 2025
abhinavdangeti
pushed a commit
that referenced
this pull request
Dec 8, 2025
…2260) - When indexing multi-vector fields (e.g., `[[3,0,0], [0,4,0]]`) with `cosine` similarity, normalization was incorrectly applied to the entire flattened array instead of each sub-vector independently, resulting in degraded similarity scores. - Added `NormalizeMultiVector(vec, dims)` that normalizes each sub-vector separately, fixing scores for multi-vector documents (e.g., score now correctly returns 1.0 instead of 0.6 for exact matches).
abhinavdangeti
pushed a commit
that referenced
this pull request
Dec 9, 2025
…2260) - When indexing multi-vector fields (e.g., `[[3,0,0], [0,4,0]]`) with `cosine` similarity, normalization was incorrectly applied to the entire flattened array instead of each sub-vector independently, resulting in degraded similarity scores. - Added `NormalizeMultiVector(vec, dims)` that normalizes each sub-vector separately, fixing scores for multi-vector documents (e.g., score now correctly returns 1.0 instead of 0.6 for exact matches).
CascadingRadium
added a commit
that referenced
this pull request
Dec 15, 2025
…2260) - When indexing multi-vector fields (e.g., `[[3,0,0], [0,4,0]]`) with `cosine` similarity, normalization was incorrectly applied to the entire flattened array instead of each sub-vector independently, resulting in degraded similarity scores. - Added `NormalizeMultiVector(vec, dims)` that normalizes each sub-vector separately, fixing scores for multi-vector documents (e.g., score now correctly returns 1.0 instead of 0.6 for exact matches).
CascadingRadium
added a commit
that referenced
this pull request
Dec 15, 2025
…2264) Author: @CascadingRadium from #2260 - When indexing multi-vector fields (e.g., `[[3,0,0], [0,4,0]]`) with `cosine` similarity, normalization was incorrectly applied to the entire flattened array instead of each sub-vector independently, resulting in degraded similarity scores. - Added `NormalizeMultiVector(vec, dims)` that normalizes each sub-vector separately, fixing scores for multi-vector documents (e.g., score now correctly returns 1.0 instead of 0.6 for exact matches). --------- Co-authored-by: Rahul Rampure <rahul.rampure@couchbase.com>
abhinavdangeti
added a commit
that referenced
this pull request
Dec 15, 2025
…ctly (#2264) Author: @CascadingRadium from #2260 - When indexing multi-vector fields (e.g., `[[3,0,0], [0,4,0]]`) with `cosine` similarity, normalization was incorrectly applied to the entire flattened array instead of each sub-vector independently, resulting in degraded similarity scores. - Added `NormalizeMultiVector(vec, dims)` that normalizes each sub-vector separately, fixing scores for multi-vector documents (e.g., score now correctly returns 1.0 instead of 0.6 for exact matches). --------- Co-authored-by: Rahul Rampure <rahul.rampure@couchbase.com>
abhinavdangeti
added a commit
that referenced
this pull request
Dec 15, 2025
…ctly (#2264) Author: @CascadingRadium from #2260 - When indexing multi-vector fields (e.g., `[[3,0,0], [0,4,0]]`) with `cosine` similarity, normalization was incorrectly applied to the entire flattened array instead of each sub-vector independently, resulting in degraded similarity scores. - Added `NormalizeMultiVector(vec, dims)` that normalizes each sub-vector separately, fixing scores for multi-vector documents (e.g., score now correctly returns 1.0 instead of 0.6 for exact matches). --------- Co-authored-by: Rahul Rampure <rahul.rampure@couchbase.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[[3,0,0], [0,4,0]]) withcosinesimilarity, normalization was incorrectly applied to the entire flattened array instead of each sub-vector independently, resulting in degraded similarity scores.NormalizeMultiVector(vec, dims)that normalizes each sub-vector separately, fixing scores for multi-vector documents (e.g., score now correctly returns 1.0 instead of 0.6 for exact matches).