Math expression null handling

[gianm]

Math expressions can run into nulls in a few situations:

1. Missing columns in expressions used for aggregations like SUM(log(x + y)). Some segments may have column x but not y.
2. Missing columns in expressions used for grouping or filtering like WHERE sin(x + y) = 1.
3. Null values in string columns, i.e. does strlen(x) return 0 or null if x is null. This can influence the value of aggregations like min(strlen(x)).
4. Missing string columns, i.e. does strlen(x) return 0 or null if the column x is missing in a segment.

We should nail down how to handle this and document that.

For (1) I think it makes sense to treat the whole expression as null if any component is null. Then the aggregation function should do something reasonable with that (such as implemented in #3627). If users want something other than this behavior, they can use nvl or if to assign default values to specific identifiers.

For (2) I think we can also treat the whole lhs expression as null and basically have the = always be false.

For (3) and (4) I think it makes sense for them to have the same behavior, and for that behavior to be that null string values don't actually exist. The identifiers should behave like empty strings. This is different from how identifiers we expect to be numeric work (which behave like nulls). That may or may not be fine?

[jaehc]

I think this could also be related to the zero-filling feature which, as far as I know, dose not fill zeros between disjoint segments at the moment, but does for only missing values in a single segment. Does this issue cover only expressions or more general idea of how to treat missing values such as the zero-filling?

[gianm]

@jaehc IMO this issue is just about null handling for math expressions. If you want different behavior for zero filling in timeseries that could be a separate issue. Mostly since the code paths are not really very related.

[jihoonson]

I wonder this issue covers the comparison of nulls and boolean values. If so, null handling in sql can be a good reference. It specifies the null propagation in arithmetic and string expressions as well. See https://en.wikipedia.org/wiki/Null_(SQL).

[gianm]

@jihoonson agree, I think something as close to SQL as possible would be good. The only situation I'd consider departing from SQL is if SQL behavior would create a really drastic discrepancy with how non-expression-based Druid queries work. Even then it could be nice to have an option to be more SQL like.

[gianm]

Bumping this to 0.10.1, along with the pr for linking expressions into the docs.

[gianm]

Resolved (for now) by #4394, which implements the following behavior:

• Missing columns are treated as null values.
• null values in string context are treated the same as empty strings, so strlen(x) is 0 if x is null.
• null values in numeric context are treated as zeroes, so x + y is 3 if x is null and y is 3.

This behavior is generally consistent with basic Druid column selectors, which will treat missing columns as null or 0 (based on the selector type).

Related: #4349, which may bring more sweeping changes.