Repro:
import pyarrow as pa
import lance
struct_type = pa.struct([("tag", pa.string())])
# Create struct array where 2nd element is null
tag_array = pa.array(["valid", "null_struct", "valid", "valid"])
struct_array = pa.StructArray.from_arrays(
[tag_array],
fields=[pa.field("tag", pa.string(), nullable=True)],
mask=pa.array([True, False, True, True]) # False = null struct element
)
# Create list containing these structs
offsets = pa.array([0, 4], type=pa.int32())
list_array = pa.ListArray.from_arrays(offsets, struct_array)
batch = pa.record_batch(
[pa.array([0]), list_array],
names=["id", "value"]
)
ds = lance.write_dataset(batch, "memory://", data_storage_version="2.2")
ds.to_table()
thread 'lance_background_thread' panicked at /Users/willjones/Documents/lance/rust/lance-encoding/src/encodings/logical/struct.rs:382:13:
assertion `left == right` failed
left: 1
right: 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ds.to_table()
~~~~~~~~~~~^^
File "/Users/willjones/Documents/lance/python/python/lance/dataset.py", line 1106, in to_table
).to_table()
~~~~~~~~^^
File "/Users/willjones/Documents/lance/python/python/lance/dataset.py", line 5135, in to_table
return self.to_reader().read_all()
~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "pyarrow/ipc.pxi", line 848, in pyarrow.lib.RecordBatchReader.read_all
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: External error: Wrapped error: task 65 panicked with message "assertion `left == right` failed\n left: 1\n right: 0", /Users/willjones/Documents/lance/rust/lance-encoding/src/decoder.rs:1803:39
Repro: