feat: merge-insert with primary key dedupe

[jackye1995]

Based on #4787

Co-authored-by: vinoyang vinoyang@apache.org

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[github-actions]

Code Review: Bloom Filter Merge-Insert Conflict Detection

This PR adds bloom filter-based conflict detection for merge-insert operations, tracking keys of newly inserted rows to detect concurrent insert conflicts.

Summary

The implementation adds:

• Protobuf definitions for KeyExistenceFilter (exact set or bloom filter)
• KeyExistenceFilterBuilder and KeyExistenceFilter types for tracking inserted keys
• Integration with Operation::Update to carry the filter
• Conflict detection in check_update_txn using filter intersection

---

P0: Critical - Inconsistent Hashing Between Bloom Filter Helper and Main SBBF

File: rust/lance-index/src/scalar/bloomfilter/sbbf.rs:428-454

The newly added bloom_contains_hash() helper function uses a completely different hashing scheme than the existing Sbbf implementation:

// New helper (lines 428-447) - custom rolling seed hash
pub fn bloom_contains_hash(hash: u64, bitmap: &[u8], num_bits: u32) -> bool {
    let mut seed = 0x9e3779b97f4a7c15u64;
    for _i in 0..SBBF_NUM_HASHES {
        let pos = ((hash.wrapping_add(seed)) % m) as usize;
        // ... bit_test on raw bytes
        seed = seed.rotate_left(13) ^ 0x517cc1b727220a95u64;
    }
}

The existing Sbbf uses XxHash64 and the Parquet spec's block-based approach with specific salt values. These two approaches are not compatible - you cannot build a filter with Sbbf::insert() and then query it with bloom_contains_hash().

Looking at the actual usage in KeyExistenceFilter::intersects(), the code calls bloom_contains_hash() on hashes from the ExactSet against a Bloom bitmap that was built using the Sbbf implementation. This will produce incorrect results (both false negatives and random behavior).

Recommendation: Either:

1. Add a proper contains_hash(hash: u64) method to the Sbbf struct that works with its internal representation, or
2. Store key hashes computed the same way Sbbf computes them in the ExactSet

---

P1: Test Coverage Gap for Cross-Filter Type Intersection

File: rust/lance/src/dataset/write/merge_insert.rs (tests section)

The tests cover Bloom-to-Bloom intersection via concurrent merge inserts, but don't explicitly test:

• ExactSet vs Bloom intersection paths (used when one transaction has few keys)
• Serialization/deserialization roundtrip with ExactSet variant

Given the P0 issue above, these paths would be broken. Consider adding explicit unit tests for KeyExistenceFilter::intersects() covering all four type combinations.

---

Minor Observations (non-blocking)

1. Dead code path: FilterType::ExactSet is defined but never constructed in the production code path - only KeyExistenceFilterBuilder::build() is used, which always produces a Bloom filter.

2. Constant sizing: DEFAULT_NUMBER_OF_ITEMS = 8192 and DEFAULT_PROBABILITY = 0.00057 are fixed. For very small inserts (e.g., 1-10 rows), this over-allocates. For very large inserts, false positive rate may be higher than expected. Consider if dynamic sizing based on actual row count is desirable.

3. Module visibility: The diff changes mod write to pub(crate) mod write in dataset.rs. Ensure this broader visibility is intentional and won't expose internal APIs unintentionally.

---

The P0 hashing inconsistency should be addressed before merge as it could cause incorrect conflict detection behavior (both missed conflicts and spurious retries).

[codecov]

Codecov Report

❌ Patch coverage is 80.35714% with 132 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ce/src/dataset/write/merge_insert/inserted_rows.rs	56.47%	81 Missing and 3 partials ⚠️
rust/lance-index/src/scalar/bloomfilter/sbbf.rs	23.68%	29 Missing ⚠️
rust/lance/src/io/commit/conflict_resolver.rs	81.13%	10 Missing ⚠️
rust/lance/src/dataset/write/merge_insert.rs	98.43%	0 Missing and 5 partials ⚠️
...lance/src/dataset/write/merge_insert/exec/write.rs	94.44%	1 Missing and 2 partials ⚠️
rust/lance/src/dataset/transaction.rs	88.88%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[yanghua]

+1