Describe the bug
Comet encountered some errors on Spark test like following. It is because DataFusion SortExec calls lexsort_to_indices and the function fallbacks to sort_to_indices for single column case. However, sort_to_indices doesn't support all data types, e.g., struct. For unsupported data types, lexsort_to_indices should not fallback to sort_to_indices.
[info] - vectorized reader: struct *** FAILED *** (515 milliseconds)
[info] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 644.0 failed 1 times, most recent failure: Lost task 0.0 in stage 644.0 (TID 862) (192.168.86.44 executor driver): org.apache.comet.CometNa
tiveException: Compute error: Sort not supported for data type Struct([Field { name: "_1", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "_2", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }])
To Reproduce
Expected behavior
Additional context
Describe the bug
Comet encountered some errors on Spark test like following. It is because DataFusion SortExec calls
lexsort_to_indicesand the function fallbacks tosort_to_indicesfor single column case. However,sort_to_indicesdoesn't support all data types, e.g., struct. For unsupported data types,lexsort_to_indicesshould not fallback tosort_to_indices.To Reproduce
Expected behavior
Additional context