Move create_physical_expr to phy-expr-common #3

[jayzhan211]

Which issue does this PR close?

Closes #10176
Closes #10074
Closes #10144

All in one in this PR.

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

[alamb]

Hi @jayzhan211 -- if we merge this one, does it close #10176 and #10144 ?

[jayzhan211]

Hi @jayzhan211 -- if we merge this one, does it close #10176 and #10144 ?

Yes!

[alamb]

Sorry for being dense -- would you like me to review this PR? Or do you think we should review (and merge) #10176 and #10144 first?

[jayzhan211]

It depends on you and other reviewers, If you don't mind the large PR, definitely it is nice we can merge this one.
Otherwise, we can close #10144 first, and move on to #10176 and finally this one.

I didn't merge #10144 because I'm also curious about what is it like at the final PR, make sure not all the things are moved to common

[alamb]

Thank you for this @jayzhan211

I spent some time reviewing this PR this morining

One thing I am missing is the big picture description / story of "what code belongs in what crate". I suspect there is a good story here but I can't quite figure it out.

In this case, I think getting the larger story straight will help avoid churn / code moving back and forth unecessairly. The larger story is also likely the key to understanding the end state.

create_physical_expr

As I understand it, this PR is part of the larger quest to move aggregate functions out of datafusion-physical-expr #8708

In my mind, create_physical_expr is a big switch statement between Expr --> Arc<dyn PhysicalExpr>

Once we have created a Arc<dyn PhysicalExpr> I don't really understand how/why we would ever get an Expr that we need to convert back to Arc<dyn PhysicalExpr> again.

End state of dependencies??

Is the eventual plan to have something like the following? If so, I don't understand why the functions-aggregates needs to depends on create_physical_erxpr

  graph TD;
      functions-aggregates-->physical-expr-common;
      physical-expr-->physical-expr-common;
      core-->functions-aggregates;
      core-->physical-expr;

Loading

[alamb]

Here is another suggestion: #10074 (comment)

[jayzhan211]

Once we have created a Arc I don't really understand how/why we would ever get an Expr that we need to convert back to Arc again.

I agree, but it turns out I didn't find a way to reuse ordering_req in AggregateFunctionExpr, because we can't have physical-expr in AggregateUDFImpl, so is not able to get it straightaway

End state of dependencies??

I agree with the graph, which is in my mind too.

[jayzhan211]

If we can move reverse_expr related logic all to logical layer.... 🤔