add untyped null

[jimexist]

Which issue does this PR close?

Closes #.

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

[Dandandan]

I am not sure anymore this is a good idea.

The problem with introducing null as scalar is that we lose the type information on the scalar (making everything type Null). In order to make this work, the value should be converted to an array of the original type afterwards.

[houqp]

That's a good point @Dandandan

losing type info

[jimexist]

I am not sure anymore this is a good idea.

The problem with introducing null as scalar is that we lose the type information on the scalar (making everything type Null). In order to make this work, the value should be converted to an array of the original type afterwards.

The alternative I think is to infer types for null during planning

[alamb]

I think that during planning we should treat the type of any ScalarValue::*(None) as DataType::Null and coerce them appropriately to the needed types as part of physical planning.

@jimexist I think added the needed casting logic in arrow-rs at apache/arrow-rs#884 (though that didn't make it into 6.1.0 -- so I suppose we have to wait a bit longer -- though maybe I'll end up making a 6.2.0 before 6.1.0 gets through the voting process 🤣 )

Once we can cast from DataType::Null I think we'll be able to make this work in DataFusion

[liukun4515]

any progress, is there any context or doc for this?
@jimexist

[alamb]

I had effectively changed Expr::Scalar to report its type as DataType::Null its interior ScalarValue::is_null() returned true. That seemed fairly promising but I got blocked by the lack of null cast kernels in Arrow.

Now I think we have those kernels it would be useful to try again -- but I am not sure when I will have the time again

[alamb]

I am trying to clean up the list of PRs to review in DataFusion so marking old ones as stale. Please let us know if you plan to work on this soon. Otherwise we will close it down and reopen it when the time is right.

[alamb]

Closing stale PRs, please reopen if this was a mistake and you plan to keep working on this one