[VL] Read OrcFile error when schema in Orc file and the table file don't consist

[liujp]

Backend

VL (Velox)

Bug description

when the schema in Orc file is different from the table schema(especially the field name), the gluten read error. as follows:
schema in ORC file:

the user table is:
CREATE TABLE parquet_test
(
treatment TINYINT,
numerator DOUBLE,
denominator TINYINT,
numerator_pre BIGINT,
denominator_pre TINYINT,
Y DOUBLE,
X1 INT,
X2 INT,
X3 INT,
X3_string STRING,
X7_needcut BIGINT,
X8_needcut BIGINT,
weight DOUBLE,
distance DOUBLE
)

then execute the query:
val df = spark.sql( "SELECT count(Y) from udf_table")
a incorrect value returns,

but use query: val df = spark.sql( "SELECT count(_col5) from udf_table")
return ok;

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

No response

[kecookier]

Low version hive-orc file not contain field name, only with field index. I will try to fix it.

[liujp]

Low version hive-orc file not contain field name, only with field index. I will try to fix it.

you are right, mainly compact for lower version. thx!

[liujp]

Low version hive-orc file not contain field name, only with field index. I will try to fix it.

velox has already support using different schema(table schema and file schema )，user need to add a schema to create hive table handle

[kecookier]

Yes, we can pass the table schema parameter (dataColumns) when creating a HiveTableHandle, but currently it's set to nullptr.

https://github.com/apache/incubator-gluten/blob/main/cpp/velox/substrait/SubstraitToVeloxPlan.cc#L1129
https://github.com/apache/incubator-gluten/blob/main/cpp/velox/substrait/SubstraitToVeloxPlan.cc#L1094

[wenwj0]

@liujp @kecookier Any solution for this? I also met this problem.

[liujp]

@liujp @kecookier Any solution for this? I also met this problem.

Now I add a schema table to ReadRelNode, and serialize it to Protobuf LocalFiles. gluten deserialize the schema in LocalFiles, and the pass the schema to HiveTableHandle. But for ORC file, this schema is only the top level, for the columns which has child type, this schema is not unabridged. For this condition, ORC reader should reuse the parquet reader.

[kecookier]

@ccat3z