Skip to content

[Rust] [DataFusion] Allow DataFusion to cast all type combinations supported by Arrow cast kernel #26173

@asfimport

Description

@asfimport

When the DataFusion planner inserts casts, today it relies on special logic to determine the valid coded casts.

The actual arrow cast kernels support a much wider range of data types, and thus DataFusion is artificially limiting the casts it supports for no particularly good reason I can see.

This ticket tracks the work to remove the extra casting checking in the datafusion planner and instead simply rely on runtime check of arrow cast compute kernel

The potential downside of this approach is that the error may be generated later in the execution process (rather than the planner), and possibly have a less specific error message, the upside is there is less code and we get several conversions immediately (like timestamp predicate casting)

I also plan to add DictionaryArray support to the casting kernels and I would like to avoid having to replicate some part of that logic in DataFusion

Reporter: Andrew Lamb / @alamb
Assignee: Andrew Lamb / @alamb

PRs and other links:

Note: This issue was originally created as ARROW-10165. Please see the migration documentation for further details.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions