Regression in in parquet `42.0.0` : Bad parquet column indexes for All Null Columns, resulting in `Parquet error: StructArrayReader out of sync` on read

[tustvold]

Describe the bug

Writing a column consisting solely of all nulls results in an empty offset index for that column

#[test]
fn test_writer_all_null() {
    let a = Int32Array::from(vec![1, 2, 3, 4, 5]);
    let b = Int32Array::new(vec![0; 5].into(), Some(NullBuffer::new_null(5)));
    let batch = RecordBatch::try_from_iter(vec![
        ("a", Arc::new(a) as ArrayRef),
        ("b", Arc::new(b) as ArrayRef),
    ])
    .unwrap();

    let mut buf = Vec::with_capacity(1024);
    let mut writer = ArrowWriter::try_new(&mut buf, batch.schema(), None).unwrap();
    writer.write(&batch).unwrap();
    writer.close().unwrap();

    let bytes = Bytes::from(buf);
    let options = ReadOptionsBuilder::new().with_page_index().build();
    let reader = SerializedFileReader::new_with_options(bytes, options).unwrap();
    let index = reader.metadata().offset_index().unwrap();

    assert_eq!(index.len(), 1);
    assert_eq!(index[0].len(), 2); // 2 columns
    assert_eq!(index[0][0].len(), 1); // 1 page
    assert_eq!(index[0][1].len(), 1); // 1 page
}

This appears to have been a bug introduced by #4389

In particular - https://github.com/apache/arrow-rs/pull/4389/files#diff-b1859e4da1d85e57a4185dc407458ac83a369dac132285689c27e878e3695ad6R695

To Reproduce

Expected behavior

Additional context

[alamb]

I added some more stuff to the title of this ticket to make it easier to find / search for

[alamb]

FYI We found this in our internal testing. I will post symptoms here to help anyone else who comes across this:

We found a query like this in IOx that resulted in Parquet error: StructArrayReader out of sync on read errors

$ datafusion-cli -c "SELECT col, time FROM 'data.parquet' WHERE 1684850057953220316 <= time::bigint"
DataFusion CLI v27.0.0
Arrow error: External error: Arrow: Parquet argument error: Parquet error: StructArrayReader out of sync in read_records, expected 0 skipped, got 11

The workaround for datafusion is to disable using the page index:

❯ set datafusion.execution.parquet.enable_page_index = false;

[cmackenzie1]

I am still experiencing this error with Datafusion 28.0.0, which is based on Arrow 43.0.0. Is it possible the bug still exists?

FWIW, I tested against a file that was written using parquet 42.0.0 and read using 43.0.0. Testing against a file that was read/written using 43.0.0 works ok it seems.