Skip to content

Lance 2.1/2.2 panics on null struct elements in List<Struct> #5930

@wjones127

Description

@wjones127

Repro:

import pyarrow as pa
import lance

struct_type = pa.struct([("tag", pa.string())])

# Create struct array where 2nd element is null
tag_array = pa.array(["valid", "null_struct", "valid", "valid"])
struct_array = pa.StructArray.from_arrays(
    [tag_array],
    fields=[pa.field("tag", pa.string(), nullable=True)],
    mask=pa.array([True, False, True, True])  # False = null struct element
)
# Create list containing these structs
offsets = pa.array([0, 4], type=pa.int32())
list_array = pa.ListArray.from_arrays(offsets, struct_array)
batch = pa.record_batch(
    [pa.array([0]), list_array],
    names=["id", "value"]
)

ds = lance.write_dataset(batch, "memory://", data_storage_version="2.2")
ds.to_table()
thread 'lance_background_thread' panicked at /Users/willjones/Documents/lance/rust/lance-encoding/src/encodings/logical/struct.rs:382:13:
assertion `left == right` failed
  left: 1
 right: 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    ds.to_table()
    ~~~~~~~~~~~^^
  File "/Users/willjones/Documents/lance/python/python/lance/dataset.py", line 1106, in to_table
    ).to_table()
      ~~~~~~~~^^
  File "/Users/willjones/Documents/lance/python/python/lance/dataset.py", line 5135, in to_table
    return self.to_reader().read_all()
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "pyarrow/ipc.pxi", line 848, in pyarrow.lib.RecordBatchReader.read_all
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: External error: Wrapped error: task 65 panicked with message "assertion `left == right` failed\n  left: 1\n right: 0", /Users/willjones/Documents/lance/rust/lance-encoding/src/decoder.rs:1803:39

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingcritical-fixBugs that cause crashes, security vulnerabilities, or incorrect data.

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions