Panic in `adjust_child_validity` when merging Null-typed arrays: "Arrays of type Null cannot contain a null bitmask"

[wjones127]

Summary

Scanning or taking rows from a dataset panics with:

thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value:
InvalidArgumentError("Arrays of type Null cannot contain a null bitmask")'
rust/lance-arrow/src/lib.rs:1187

Steps to Reproduce

The panic occurs when all three conditions are met:

1. A dataset has a struct column (e.g. meta: struct<name: Utf8, extra: Null>) where the extra sub-field has DataType::Null — this happens when Arrow infers the type for a column that contains only null values (e.g. from a Python/pandas None-filled column).
2. A subsequent fragment is appended that omits the nullable extra sub-field (permitted by allow_missing_if_nullable).
3. A scan or take operation reads the second fragment — Lance adds a NullReader for the missing extra: Null sub-field.

Minimal Rust reproducer:

// Fragment 0: struct column with a DataType::Null sub-field
// (simulates inserting rows where a column has all-null values and Arrow infers DataType::Null)
let meta0 = StructArray::new(
    Fields::from(vec![
        Field::new("name", DataType::Utf8, true),
        Field::new("extra", DataType::Null, true),
    ]),
    vec![
        Arc::new(StringArray::from(vec![Some("alice"), Some("bob")])) as ArrayRef,
        Arc::new(NullArray::new(2)) as ArrayRef,
    ],
    None,
);

// Fragment 1: same struct but without `extra`; some rows have a null struct
let meta1 = StructArray::new(
    Fields::from(vec![Field::new("name", DataType::Utf8, true)]),
    vec![Arc::new(StringArray::from(vec![Some("charlie"), None])) as ArrayRef],
    Some(vec![true, false].into()), // row 1 is a null struct
);

// ... write both fragments, then scan ...
// Panics: "Arrays of type Null cannot contain a null bitmask"

Even simpler, at the merge level:

let left = StructArray::new(
    Fields::from(vec![Field::new("a", DataType::Int32, true)]),
    vec![Arc::new(Int32Array::from(vec![Some(1), None])) as ArrayRef],
    Some(vec![true, false].into()),
);
let right = StructArray::new(
    Fields::from(vec![Field::new("b", DataType::Null, true)]),
    vec![Arc::new(NullArray::new(2)) as ArrayRef],
    Some(vec![true, false].into()),
);
merge(&left, &right); // panics

Root Cause

adjust_child_validity (lance-arrow/src/lib.rs) propagates a parent struct's null validity down to child arrays. When the child has DataType::Null, it attempts:

ArrayData::try_new(
    DataType::Null,
    len,
    Some(null_buffer), // <-- Arrow rejects this
    ...
)
.unwrap() // panics

Arrow's DataType::Null type cannot have an explicit null bitmap — all-null is implied by the type itself. The code does not check for this before calling try_new.

Call Stack

lance_arrow::adjust_child_validity
lance_arrow::merge
lance_arrow::merge                         ← recursive call into sub-struct
<RecordBatch as RecordBatchExt>::merge
lance_table::utils::stream::MergeStream::emit
...
lance::dataset::take::do_take_rows

The double merge frame indicates the panic occurs when recursing into a nested struct: the parent struct has non-zero null count, and a child column in that struct has DataType::Null.

Fix

In adjust_child_validity, add an early return when child.data_type() == &DataType::Null. A Null array is always entirely null by definition; no validity adjustment is needed or possible.

if child.data_type() == &DataType::Null {
    return child.clone();
}

Impact

Any query that goes through MergeStream (scans, takes, vector search result fetches) will panic for the affected fragments, making the table completely unreadable until the data is compacted or the fragment is otherwise removed.