Describe the bug
In #651, Comet encountered some errors on Spark test like following. It is because DataFusion SortExec calls lexsort_to_indices and the function fallbacks to sort_to_indices for single column case. However, sort_to_indices doesn't support all data types, e.g., struct. For unsupported data types, lexsort_to_indices should not fallback to sort_to_indices.
[info] - vectorized reader: struct *** FAILED *** (515 milliseconds)
[info] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 644.0 failed 1 times, most recent failure: Lost task 0.0 in stage 644.0 (TID 862) (192.168.86.44 executor driver): org.apache.comet.CometNa
tiveException: Compute error: Sort not supported for data type Struct([Field { name: "_1", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "_2", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }])
I'm going to fix it at arrow-rs side: apache/arrow-rs#6226. But before we have new releases, we need to fallback to Spark for unsupported cases like struct.
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response
Describe the bug
In #651, Comet encountered some errors on Spark test like following. It is because DataFusion SortExec calls
lexsort_to_indicesand the function fallbacks tosort_to_indicesfor single column case. However,sort_to_indicesdoesn't support all data types, e.g., struct. For unsupported data types,lexsort_to_indicesshould not fallback tosort_to_indices.I'm going to fix it at arrow-rs side: apache/arrow-rs#6226. But before we have new releases, we need to fallback to Spark for unsupported cases like struct.
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response