Add `FunctionRewrite` API, Move Array specific rewrites to `datafusion_functions_array` #9583

alamb · 2024-03-12T16:11:57Z

Which issue does this PR close?

Closes #9519
Closes #9557

This is an alternate to the design @jayzhan22 implemented in #9557

Rationale for this change

While reviewing XX I realized that the suggestion to use AnalyzerRules in the original ticket was somewhat overkill. The core function needed is rewriting Exprs (operators specifically) to Functions.

Using AnalyzerRule requires an entire LogicalPlan

What changes are included in this PR?

Formalizes the general pattern of rewriting certain expressions / operators to function calls, specifically:

Add a new FunctionRewrite trait
Add an API to register FunctionRewriters with FunctionRegistry
Demonstrate the use of this API by porting the array specific rewrite as a FunctionRewriter

Are these changes tested?

Covered by existing tests

Are there any user-facing changes?

There is a new API that users can use to rewrite DataFusion Exprs to their own function calls.

…sion-array-functions crate

alamb · 2024-03-12T16:12:59Z

datafusion/expr/src/expr_rewriter/mod.rs

+/// For example, concatenating arrays `a || b` is represented as
+/// `Operator::ArrowAt`, but can be implemented by calling a function
+/// `array_concat` from the `functions-array` crate.
+pub trait FunctionRewrite {


This is the new trait used to rewrite Exprs to function calls

alamb · 2024-03-12T16:14:23Z

datafusion/functions-array/src/rewrite.rs

+        schema: &DFSchema,
+        _config: &ConfigOptions,
+    ) -> datafusion_common::Result<Transformed<Expr>> {
+        let transformed = match expr {


The whole point of this PR is to move these rules out of datafusion/optimizer/src/analyzer/rewrite_expr.rs in the optimizer crate and into the functions-array crate

After reading this PR, a question for me is in #9563, get_field function also works for a single ScalarValue of Struct or Map
Should I put the rewrite logic in this file or create a new rewrite file under "datafusion/functions"? 🤔

I suggest we add another match in this same file

We could also add another FunctionRewriter but that would result in another (unecessary walk down the tree I think)

Yeah, make sense. I thought it's OK for function but may not suitable for semantic, but now adding in the same file works, we may separate another file if there're more function_rewrite.

alamb · 2024-03-12T16:14:51Z

datafusion/functions-array/src/rewrite.rs

+                    && is_func(&left, "make_array")
+                    && is_func(&right, "make_array") =>
+            {
+                Transformed::yes(array_has_all(*left, *right))


I also rewrote these rules to avoid cloneing (as they all get an owned Expr)

alamb · 2024-03-12T16:17:46Z

datafusion/optimizer/src/analyzer/function_rewrite.rs

@@ -0,0 +1,123 @@
+// Licensed to the Apache Software Foundation (ASF) under one


Github renders this as a new file, but it is the old datafusion/optimizer/src/analyzer/rewrite_expr.rs with the array specific logic removed and instead invokes each FunctionRewrite individually

alamb · 2024-03-12T16:18:19Z

datafusion/optimizer/src/analyzer/mod.rs

-pub mod rewrite_expr;
-pub mod subquery;
-pub mod type_coercion;
+use std::sync::Arc;


I don't know why the imports were moved -- I think my editor did it

datafusion/functions-array/src/rewrite.rs

+    ) -> datafusion_common::Result<Transformed<Expr>> {
+        let transformed = match expr {
+            // array1 @> array2 -> array_has_all(array1, array2)
+            Expr::BinaryExpr(BinaryExpr { left, op, right })


jayzhan211 · 2024-03-13T00:44:42Z

datafusion/optimizer/src/analyzer/function_rewrite.rs

+        plan.with_new_exprs(new_expr, new_inputs)
+    }
+}
+struct OperatorToFunctionRewriter<'a> {


references are added for performance?

Yes I think so. In order to pass down the ConfigOptions without clone'ing the them specifically

I think in general we could improve planning efficiency in DataFusion by avoiding clone but that is a project for another day

jayzhan211

It looks really nice!

alamb

Thank you for the review @jayzhan211

I am going to merge this in to unblock more function migration work

alamb · 2024-03-13T15:11:15Z

datafusion/functions-array/src/rewrite.rs

+        schema: &DFSchema,
+        _config: &ConfigOptions,
+    ) -> datafusion_common::Result<Transformed<Expr>> {
+        let transformed = match expr {


I suggest we add another match in this same file

We could also add another FunctionRewriter but that would result in another (unecessary walk down the tree I think)

alamb · 2024-03-13T15:14:07Z

datafusion/optimizer/src/analyzer/function_rewrite.rs

+        plan.with_new_exprs(new_expr, new_inputs)
+    }
+}
+struct OperatorToFunctionRewriter<'a> {


Yes I think so. In order to pass down the ConfigOptions without clone'ing the them specifically

I think in general we could improve planning efficiency in DataFusion by avoiding clone but that is a project for another day

alamb added 2 commits March 12, 2024 12:07

Add FunctionRewrite API, port array specific rewrites to the datafu…

c268cd9

…sion-array-functions crate

remove unecessary cfg

96d256e

github-actions bot added logical-expr Logical plan and expressions optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Mar 12, 2024

alamb commented Mar 12, 2024

View reviewed changes

alamb changed the title ~~Add FunctionRewrite API, port array specific rewrites to the datafu…~~ Add FunctionRewrite API, Move Array specific rewrites to datafusion_functions_array Mar 12, 2024

This was referenced Mar 12, 2024

Move Array specific rewrites to datafusion_functions_array AnalyzerRule #9557

Closed

Remove physical expr of NamedStructField, convert to get_field function call #9563

Merged

jayzhan211 reviewed Mar 13, 2024

View reviewed changes

datafusion/functions-array/src/rewrite.rs

) -> datafusion_common::Result<Transformed<Expr>> {

let transformed = match expr {

// array1 @> array2 -> array_has_all(array1, array2)

Expr::BinaryExpr(BinaryExpr { left, op, right })

This comment was marked as resolved.

Sign in to view

jayzhan211 reviewed Mar 13, 2024

View reviewed changes

jayzhan211 approved these changes Mar 13, 2024

View reviewed changes

alamb commented Mar 13, 2024

View reviewed changes

alamb merged commit a43938d into apache:main Mar 13, 2024

alamb deleted the alamb/function_rewrite branch March 13, 2024 17:37

alamb mentioned this pull request Apr 10, 2024

Regression: Error in NamedStructField should be rewritten in OperatorToFunction with subquery #10029

Closed

jayzhan211 mentioned this pull request May 11, 2024

Move optimizer rule that has aggregate function out of core #10455

Closed

		@@ -0,0 +1,123 @@
		// Licensed to the Apache Software Foundation (ASF) under one

Add FunctionRewrite API, Move Array specific rewrites to datafusion_functions_array #9583

Add FunctionRewrite API, Move Array specific rewrites to datafusion_functions_array #9583

Uh oh!

Conversation

alamb commented Mar 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yyy1000 Mar 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jayzhan211 left a comment

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add `FunctionRewrite` API, Move Array specific rewrites to `datafusion_functions_array` #9583

Add `FunctionRewrite` API, Move Array specific rewrites to `datafusion_functions_array` #9583

alamb commented Mar 12, 2024 •

edited

Loading

yyy1000 Mar 13, 2024 •

edited

Loading