[Rust][DataFusion] Improve performance of equality to a constant predicate support

[asfimport]

I noticed this behavior while working on support for DictionaryArrays and wanted to capture it in a ticket in case someone has time to work on it.

In order to implement an equality predicate to a constant such as d1 = 'three', DataFusion effectively creates an array with the same value 'three' repeated over and over again and uses the equality compute kernel. This is ... suboptimal.

Here is what the predicate looks like:

        predicate: BinaryExpr {
            left: CastExpr {
                expr: Column {
                    name: "d1",
                },
                cast_type: Utf8,
            },
            op: Eq,
            right: Literal {
                value: Utf8("three"),
            },
        },

Reporter: Andrew Lamb / @alamb
Assignee: Yordan Pavlov / @yordan-pavlov

Related issues:

• [Rust] implement scalar comparison operations (relates to)
• [Rust][DataFusion] improve performance of building literal arrays (relates to)

PRs and other links:

• GitHub Pull Request #8660

_{Note: This issue was originally created as ARROW-10173. Please see the migration documentation for further details.}

[asfimport]

Yordan Pavlov / @yordan-pavlov:
this issue is related to:

https://issues.apache.org/jira/browse/ARROW-8908 

which proposes an optimization to creating literal arrays; 

this, however, is still sub-optimal; ideally comparison to literal values should be performed directly as proposed here:

https://issues.apache.org/jira/browse/ARROW-8907

 

[asfimport]

Andrew Lamb / @alamb:
I agree that implementing direct comparison of array to scalar is the fastest way to make such predicates fast

[asfimport]

Yordan Pavlov / @yordan-pavlov:
I have an initial implementation of direct comparison operations to scalar values in datafusion which, for the simple query used in the benchmark ("select f32, f64 from t where f32 >= 250 and f64 > 250") shows approximately 10x performance improvement:

before:
filter_scalar time: [35.733 ms 36.613 ms 37.924 ms]

after:
filter_scalar time: [3.5938 ms 3.6450 ms 3.7035 ms]
change: [-90.048% -89.846% -89.625%] (p = 0.00 < 0.05)

 

I have also added a benchmark to compare the change in performance when comparing two arrays (using query "select f32, f64 from t where f32 >= f64") and it is negligible:

before:
filter_array time: [11.601 ms 11.656 ms 11.718 ms]

after:
filter_array time: [11.854 ms 11.957 ms 12.070 ms]
change: [+1.8032% +3.6391% +5.5671%] (p = 0.00 < 0.05)

I will be submitting a PR for this change soon.

[asfimport]

Andrew Lamb / @alamb:
@yordan-pavlov this sounds like great news. I can't wait to see it

[asfimport]

Jorge Leitão / @jorgecarleitao:
Issue resolved by pull request 8660
#8660