Is your feature request related to a problem or challenge?
Union arrays currently cannot be used in comparison operations with scalar values or other types. When attempting to filter or compare a union column against a constant value (e.g., where union_column = 67), datafusion fails during type coercion
This prevents common filtering and selection ops on union-typed data. At work, we interact with a lot of JSON and accessing data with -> returns Union types. Currently the following query will err:
select * from records
where json_val->'id' = 123
Describe the solution you'd like
- Write a
union_coercion function that handles union-to-scalar and union-to-union type coercion, following the pattern established by other composite types like struct_coercion and map_coercion
- Extend
comparison_coercion to chain to union_coercion as part of its coercion attempts
On the arrow-side, comparison kernel support for unions are being developed:
Is your feature request related to a problem or challenge?
Union arrays currently cannot be used in comparison operations with scalar values or other types. When attempting to filter or compare a union column against a constant value (e.g.,
where union_column = 67), datafusion fails during type coercionThis prevents common filtering and selection ops on union-typed data. At work, we interact with a lot of JSON and accessing data with
->returns Union types. Currently the following query will err:Describe the solution you'd like
union_coercionfunction that handles union-to-scalar and union-to-union type coercion, following the pattern established by other composite types likestruct_coercionandmap_coercioncomparison_coercionto chain tounion_coercionas part of its coercion attemptsOn the arrow-side, comparison kernel support for unions are being developed:
cmpkernel arrow-rs#8837