[Bug] BTREE index incorrect row count on OR predicate with NULLs

[fenfeng9]

Description

When using a BTREE scalar index, an OR filter involving NULL values returns a different row count compared to a full scan.
The scalar index path incorrectly handles three-valued logic when combining AllowList and BlockList masks, causing NULL rows to be incorrectly included in the result set.

How to reproduce

import tempfile

import lance
import pyarrow as pa

# One NULL and one non-NULL value
data = pa.table({"c1": pa.array([None, 1], type=pa.int64())})
filter_expr = "(c1 != 0) OR (c1 < 5)"

with tempfile.TemporaryDirectory() as d:
    lance.write_dataset(data, f"{d}/no")
    lance.write_dataset(data, f"{d}/ix")
    ds_no = lance.dataset(f"{d}/no")
    ds_ix = lance.dataset(f"{d}/ix")
    ds_ix.create_scalar_index("c1", index_type="BTREE")

    print("no_index:", ds_no.to_table(filter=filter_expr).num_rows)
    print("with_index:", ds_ix.to_table(filter=filter_expr).num_rows)

Result

no_index: 1
with_index: 2

[fenfeng9]

Likely in NullableRowAddrMask OR logic (AllowList | BlockList).
See

lance/rust/lance-core/src/utils/mask/nullable.rs

Lines 238 to 248 in 556851b

	(Self::AllowList(allow), Self::BlockList(block))
	| (Self::BlockList(block), Self::AllowList(allow)) => {
	let nulls = if allow.nulls.is_empty() && block.nulls.is_empty() {
	RowAddrTreeMap::new() // Fast path
	} else {
	// null or null -> null (excluding rows that are true in either)
	let allow_true = allow.selected.clone() - &allow.nulls;
	((allow.nulls | block.nulls) & block.selected.clone()) - allow_true
	};
	let selected = (block.selected - allow.selected) | &nulls;
	Self::BlockList(NullableRowAddrSet { selected, nulls })

[fenfeng9]

Two unit tests in rust/lance-core/src/utils/mask/nullable.rs to reproduce:

#[test]
fn test_or_allow_block_keeps_block_nulls() {
    // allow: TRUE=[1], NULL=[0]; block: FALSE=[], NULL=[0]
    let allow_mask = allow(&[1], &[0]);
    let block_mask = block(&[], &[0]);
    let result = allow_mask | block_mask;

    // Row 1 is TRUE; row 0 remains NULL (not selected)
    assert_mask_selects(&result, &[1], &[0]);
}

#[test]
fn test_or_allow_block_keeps_block_nulls_with_false_rows() {
    // allow: TRUE=[2], NULL=[]; block: FALSE=[1], NULL=[0]
    let allow_mask = allow(&[2], &[]);
    let block_mask = block(&[1], &[0]);
    let result = allow_mask | block_mask;

    // Row 2 is TRUE; row 1 is FALSE; row 0 remains NULL (not selected)
    assert_mask_selects(&result, &[2], &[0, 1]);
}

[westonpace]

This is a good find. I agree that's the wrong result for those OR operations.