UDFs marked as volatile do not appear to evaluate multiple times for each output row.

[dadepo]

Describe the bug

It seems a UDF with no arguments are only called once, even if signature is defined with Volatility::Volatile and also queried in the context of a table with multiple rows.

By this I mean, for example: select random_udf() from many_rows_table

There is a minimal repro here https://github.com/dadepo/df-repro

When ran, the output could be:

+-------+------+-----+-------+
| index | uint | int | float |
+-------+------+-----+-------+
| 1     | 2    | -2  | 1.0   |
| 2     | 3    | 3   | 3.3   |
| 3     |      |     |       |
+-------+------+-----+-------+
+---------------+
| random_normal |
+---------------+
| 25.0          |
| 25.0          |
| 25.0          |
+---------------+

In the run above, the 25.0 is the result of the udf and it is repeated.

To Reproduce

A minimal reproduction can be found here https://github.com/dadepo/df-repro

Expected behavior

The UDF to be evaluated per each row output.

Additional context

No response

[alamb]

I agree this is a bug. Thank you for the report @dadepo

Maybe @guojidan you have time to look into this issue as you have been looking at UDFs recently 🙏

[guojidan]

I will handle this issue 😄

[alamb]

cc @viirya as he fixed #8518

In fact, now that I write this I wonder if the issue @dadepo is hitting is actually the same as #8518 (which is not yet released, but should be in the next few days #8863 ) 🤔

[dadepo]

In fact, now that I write this I wonder if the issue @dadepo is hitting is actually the same as #8518 (which is not yet released, but should be in the next few days #8863 ) 🤔

@alamb Could be. My issue is observed in UDFs, while the other ticket, the issue is seen in native functions. It could very well be the fix in the planned release fixes this also. I can actually confirm this in a bit.

[dadepo]

@alamb I updated the repo to depend on main branch, see dadepo/df-repro@38acb9f

But when I ran it, I still got an output that indicates the issue still persists.

+-------+------+-----+-------+
| index | uint | int | float |
+-------+------+-----+-------+
| 1     | 2    | -2  | 1.0   |
| 2     | 3    | 3   | 3.3   |
| 3     |      |     |       |
+-------+------+-----+-------+
+---------------+
| random_normal |
+---------------+
| 38.0          |
| 38.0          |
| 38.0          |
+---------------+

[dadepo]

@alamb I updated to include native random function dadepo/df-repro@00feaed

and running that I get

+---------------+--------------------+
| random_normal | native_random      |
+---------------+--------------------+
| 82.0          | 0.2617618435183655 |
| 82.0          | 0.3157177219693452 |
| 82.0          | 0.3350644908486391 |
+---------------+--------------------+

Which indicates that, this works fine with native random function.

Could it also be the way UDF's work? Because from my understanding when a udf has no arguments, when called, the arguments it is defined with is set to:

 [
    NullArray(1),
]

ie one Array, regardless of the number of rows in the table. Could this be a factor?

[viirya]

It is not because if it is volatile or not.

A ScalarUDF which is defined as no argument one or all scalar inputs, DataFusion will assume its output is a scalar (you also return a one element array from random_normal UDF, but even you don't, DataFusion will take 0-index element from it to create a ScalarValue as output).

That's why random_normal returns same value for all input rows in the batch (because it is treated as a scala udf returns a scalar value for all input rows).

[viirya]

If you want random_normal to not output scalar for all input rows, simply add an input argument e.g., index column to it.

[dadepo]

If you want random_normal to not output scalar for all input rows, simply add an input argument e.g., index column to it.

But this means, passing an argument to a UDF, not because the function needs it, but because you want the UDF to be evaluated per output row in the query? This sounds like a limitation.

The native function did not require passing such an argument for it to be evaluated per output row, I was expecting there would be away to define a UDF with this behaviour.

[viirya]

Well, if you don't use make_scalar_function (The behavior I described above is defined here) to create the scalar function, you can define a ScalarUDF which can do what you want. For example, you can implement the UDF by implementing ScalarUDFImpl.

[viirya]

Or you want to stick with function definition instead of ScalarUDFImpl, you can also directly define your UDF in ScalarFunctionImplementation instead of using make_scalar_function.

[alamb]

Maybe we need to update the code that invokes a scalar UDF and if it is marked as volatile it needs to have its arguments expanded into an array before being invoked 🤔 (or know enough information so it can do that expansion itself)