[Rust] [Parquet] ArrowReader fails on some timestamp types

[asfimport]

I discovered this bug with this query

> SELECT tpep_pickup_datetime FROM taxi LIMIT 1;
General("InvalidArgumentError(\"column types must match schema types, expected Timestamp(Microsecond, None) but found UInt64 at column index 0\")") 

The parquet reader detects this schema when reading from the file:

Schema { 
  fields: [
    Field { name: "tpep_pickup_datetime", data_type: Timestamp(Microsecond, None), nullable: true, dict_id: 0, dict_is_ordered: false }
  ], 
  metadata: {} 
} 

The struct array read from the file contains:

[PrimitiveArray<UInt64>
[
  1567318008000000,
  1567319357000000,
  1567320092000000,
  1567321151000000, 

 When the Parquet arrow reader creates the record batch, the following validation logic fails:

for i in 0..columns.len() {
    if columns[i].len() != len {
        return Err(ArrowError::InvalidArgumentError(
            "all columns in a record batch must have the same length".to_string(),
        ));
    }
    if columns[i].data_type() != schema.field(i).data_type() {
        return Err(ArrowError::InvalidArgumentError(format!(
            "column types must match schema types, expected {:?} but found {:?} at column index {}",
            schema.field(i).data_type(),
            columns[i].data_type(),
            i)));
    }
}
 

Reporter: Andy Grove / @andygrove
Assignee: Renjie Liu / @liurenjie1024

Related issues:

• [Rust] [Parquet] Add support for writing temporal types (relates to)

_{Note: This issue was originally created as ARROW-8258. Please see the migration documentation for further details.}

[asfimport]

Andy Grove / @andygrove:
@liurenjie1024 [~sunchao] I may need some help with this one.

[asfimport]

Renjie Liu / @liurenjie1024:
I think the root cause is here https://github.com/apache/arrow/blob/master/rust/parquet/src/arrow/array_reader.rs#L220

The array reader only did conversion of data buffer, but left data type incorrect. I'll submit a PR to fix it this week.

[asfimport]

Neville Dipale / @nevi-me:
We'll get this fixed as part of the Parquet <> Arrow IO work

[asfimport]

Neville Dipale / @nevi-me:
I'm closing this, as the reader in the current master supports converting types correctly.