Skip to content

lexsort_to_indices should not fallback to non-lexical sort if the datatype is not supported #6226

@viirya

Description

@viirya

Describe the bug

Comet encountered some errors on Spark test like following. It is because DataFusion SortExec calls lexsort_to_indices and the function fallbacks to sort_to_indices for single column case. However, sort_to_indices doesn't support all data types, e.g., struct. For unsupported data types, lexsort_to_indices should not fallback to sort_to_indices.

[info] - vectorized reader: struct *** FAILED *** (515 milliseconds)        
[info]   org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 644.0 failed 1 times, most recent failure: Lost task 0.0 in stage 644.0 (TID 862) (192.168.86.44 executor driver): org.apache.comet.CometNa
tiveException: Compute error: Sort not supported for data type Struct([Field { name: "_1", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "_2", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }])                                                                                                                                                                                    

To Reproduce

Expected behavior

Additional context

Metadata

Metadata

Assignees

No one assigned

    Labels

    arrowChanges to the arrow cratebug

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions