fix: #3722 - Optimize LSM Vector Index fallback scan to target specific bucket instead of entire type

[lekmaneb]

Fix: Scoped vector index fallback scan to specific bucket (#3722)

Description

This PR fixes an issue in LSMVectorIndex where the fallback mechanism for recovering missing vectors would incorrectly scan all documents of a given type rather than restricting the scan to the specific bucket associated with the index.

Previously, when the number of page-parsed vectors fell significantly short of the document count, the fallback relied on database.countType() and database.scanType(). If a type contained multiple buckets, this resulted in inaccurate counts and unnecessary scanning of documents outside the index's scope.

The implementation now correctly leverages metadata.associatedBucketId to retrieve the target bucket, replacing type-wide operations with database.countBucket() and database.scanBucket().

Changes Made

• Replaced getTypeName() usage with metadata.associatedBucketId to resolve the specific target bucket.
• Updated document counting logic to use database.countBucket(bucket.getName()) instead of database.countType().
• Updated the fallback document scan to use database.scanBucket(bucket.getName(), ...) to ensure only documents within the indexed bucket are scanned for missing vectors.

Related Issues

• Fixes Missing images when vector neighbors query #3722

by Gemini

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

_{TIP This summary will be updated as you push new changes. Give us feedback}

[gemini-code-assist]

Code Review

This pull request refactors the vector index rebuilding logic to use bucket-based operations instead of type-based operations. The changes replace type-name lookups with bucket ID lookups for counting and scanning documents. Review feedback suggests removing a condition that could prevent recovery during total page corruption and optimizing the document count retrieval by calling the bucket object directly while ensuring a null check is performed.

[gemini-code-assist]

The condition !ridToLatestVector.isEmpty() prevents the fallback mechanism from triggering if the page parser fails to recover any vectors at all (e.g., due to severe corruption). Since the docCount check already handles the case where the bucket is empty, this extra check is unnecessary and prevents recovery in cases of total page corruption.

Suggested change

	if (metadata.associatedBucketId != -1 && !ridToLatestVector.isEmpty()) {
	if (metadata.associatedBucketId != -1) {

[gemini-code-assist]

Using database.countBucket(bucket.getName()) is less efficient than calling bucket.count() directly on the bucket object. Also, there is a risk of NullPointerException if bucket is null. While the surrounding try-catch block handles exceptions, a null check is preferred for clarity and to avoid unnecessary exception overhead.

Suggested change

	final com.arcadedb.engine.Bucket bucket = database.getSchema().getBucketById(metadata.associatedBucketId);
	final long docCount = database.countBucket(bucket.getName());
	if (ridToLatestVector.size() < docCount * 8 / 10) {
	final com.arcadedb.engine.Bucket bucket = database.getSchema().getBucketById(metadata.associatedBucketId);
	final long docCount = bucket != null ? bucket.count() : 0;
	if (bucket != null && ridToLatestVector.size() < docCount * 8 / 10) {

[Copilot]

Pull request overview

This PR fixes LSMVectorIndex’s “page-parse deficit” fallback path so it only counts/scans records from the bucket actually associated with the vector index (instead of scanning all buckets of a type), addressing incorrect counts and unnecessary scanning in multi-bucket types (issue #3722).

Changes:

• Resolve the target bucket via metadata.associatedBucketId rather than getTypeName().
• Replace database.countType(...) with database.countBucket(...) for the cross-check heuristic.
• Replace database.scanType(...) with database.scanBucket(...) for the fallback recovery scan.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The change scopes the fallback cross-check/scan to metadata.associatedBucketId, which fixes the multi-bucket type behavior described in #3722, but there’s no regression test covering the scenario where a type has multiple buckets and the vector index is bound to only one bucket. Please add a test that creates a type with 2+ buckets, builds a bucket-scoped LSM vector index, triggers the page-parser-missed-vectors fallback, and asserts only records from the associated bucket are scanned/used (records from the other bucket must not affect the docCount heuristic or be added to ridToLatestVector).

[ExtReMLapin]

Claude code : to the 🗑️

Gemini CLI : to the 🔝

[lvca]

I got this PR and changed little things and added an additional test: 89da329. Thanks @lekmaneb !

[codecov]

Codecov Report

❌ Patch coverage is 75.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 65.02%. Comparing base (aeb3cff) to head (ce31ade).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
...java/com/arcadedb/index/vector/LSMVectorIndex.java	75.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3775      +/-   ##
==========================================
- Coverage   65.17%   65.02%   -0.16%     
==========================================
  Files        1580     1580              
  Lines      116263   116275      +12     
  Branches    24658    24659       +1     
==========================================
- Hits        75775    75606     -169     
- Misses      30193    30355     +162     
- Partials    10295    10314      +19     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.