Move First Value UDAF and builtin first / last function to `aggregate-functions` #9960

jayzhan211 · 2024-04-05T12:45:55Z

Which issue does this PR close?

Part of #8708

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

jayzhan211 · 2024-04-05T14:02:49Z

I think the next step is to make last value udaf, so we can replace builtin first/last.

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

alamb

Thank you @jayzhan211 -- this is looking sweet and close

Can we add a re-export in of this crate in datafusion (like https://github.com/apache/arrow-datafusion/blob/c8584557cdfa7c138ab9039ceac31323f48a44d3/datafusion/core/src/lib.rs#L530-L533)?

Otherwise I think this is basically ready to go. 🚀

alamb · 2024-04-05T18:54:03Z

Cargo.toml

 ctor = "0.2.0"
 dashmap = "5.4.0"
 datafusion = { path = "datafusion/core", version = "37.0.0", default-features = false }
+datafusion-aggregate-functions = { path = "datafusion/aggregate-functions", version = "37.0.0" }


What do you think about calling the new crate datafusion-functions-aggregates (to mirror the naming of datafusion-functions and datafusion-functions-array)?

alamb · 2024-04-05T18:55:42Z

datafusion/core/src/execution/context/mod.rs

-                panic!("Failed to register UDAF: {}", err);
-            }
-        }
+        datafusion_aggregate_functions::register_all(&mut new_self)


alamb · 2024-04-05T18:56:15Z

datafusion/proto/tests/cases/roundtrip_logical_plan.rs

            lit(1),
        ),
        array_replace_all(make_array(vec![lit(1), lit(2), lit(3)]), lit(2), lit(4)),
+        // TODO: Add first value after built-in functions are deprecated


I don't understand why we would wait until the built in functions are deprecated 🤔

You are right. Sorry for leaving this confusing comment. I should have left it in the review comment, not in the code 😆

I plan to support the expr API with arguments like distinct, and ordering after I deprecated all the built-in functions.

However, I added a simple example with lit(1).

Upd: It seems I need to support expr with arguments before deprecated builtin

datafusion/aggregate-functions/src/first_last.rs

alamb · 2024-04-05T19:00:55Z

datafusion/aggregate-functions/src/first_last.rs

-    acc_args: AccumulatorArgs,
-) -> Result<Box<dyn Accumulator>> {
-    let mut all_sort_orders = vec![];
+/// TO BE DEPRECATED: Builtin FIRST_VALUE physical aggregate expression


I don't understand this comment -- what does it mean to be deprecated? What is the alternative?

Is the idea it will be replaced with a UDF version ?

I think it may be replaced with udaf

datafusion/aggregate-functions/src/macros.rs

datafusion/expr/src/utils.rs

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

jayzhan211 · 2024-04-06T06:41:59Z

Besides addressing comments, I also move AggregateFunctionExpr and create_aggregate_expr to functions-aggregate from physical-expr-common because I think they are specific to aggregate.

alamb

Thank you @jayzhan211 -- I think this looks really cool and could be merged.

I think it would be better to leave AggregateFunctionExpr and create_aggregate_expr in the physical-expr-core crate for reasons I mentioned but if you disagree or want to put them back as a follow on PR I think that would be ok too.

alamb · 2024-04-06T08:57:35Z

datafusion/functions-aggregate/src/first_last.rs

+        let mut all_sort_orders = vec![];
+
+        // Construct PhysicalSortExpr objects from Expr objects:
+        let mut sort_exprs = vec![];


alamb · 2024-04-06T09:00:52Z

datafusion/functions-aggregate/src/lib.rs

+/// Any implementation of this trait also needs to implement the
+/// `PartialEq<dyn Any>` to allows comparing equality between the
+/// trait objects.
+pub trait AggregateExpr: Send + Sync + Debug + PartialEq<dyn Any> {


I also move AggregateFunctionExpr and create_aggregate_expr to functions-aggregate from physical-expr-common

While I agree that these two structures are aggregate specific, I still think they should be in datafusion-physical-expr-common because that would allow people to use DataFusion and create their own aggregate functions without having to take DataFusion's built in aggregate functions

I see real value in keeping the APIs required to use datafusion separate from the actual implementations

I agree that we should avoid the user pulling out unnecessary things for them. I thought they would need builtin functions, but they probably don't. I will keep them in common

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

jayzhan211 · 2024-04-07T02:25:07Z

Thanks for your review @alamb.

jayzhan211 added 21 commits April 3, 2024 09:31

backup

0eaf289

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

move PhysicalExpr

5338f61

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

cleanup

450ae4b

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

move physical sort

3624964

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

cleanup dependencies

835f147

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

add readme

c5d80c8

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

disable doc test

7851de7

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

move column

f5aafb3

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

fmt

7bfc074

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

move aggregatexp

675d2fe

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

move other two utils

5220087

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

license

113a000

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

switch to ignore

fea87e3

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

move reverse order

06d87bc

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

rename to common

26e5782

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

cleanup

26f852c

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

Merge branch 'physical-expr-core' into move-agg-crate-2

65bf4a1

backup

2bc58c1

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

Merge branch 'physical-expr-core' into move-agg-crate-2

ae9db96

move acc to first value

30d5576

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

move builtin expr too

672edc7

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

github-actions bot added logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates core Core DataFusion crate labels Apr 5, 2024

jayzhan211 added 6 commits April 5, 2024 21:18

use macro

109b790

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

fmt

87d589f

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

fix doc

398e4e2

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

add todo

04c7f5e

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

rm comments

01a1ddf

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

rm unused

4871414

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

jayzhan211 added 2 commits April 5, 2024 21:54

rm unused code

1ef212b

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

change to private

b6d53a5

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

fix lock

9aa15a2

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

jayzhan211 marked this pull request as ready for review April 5, 2024 14:49

jayzhan211 requested review from alamb and mustafasrepo April 5, 2024 15:00

alamb reviewed Apr 5, 2024

View reviewed changes

This was referenced Apr 5, 2024

DataFusion weekly project plan (Andrew Lamb) - April 1, 2024 #9899

Closed

Minor: Update library documentation with new crates #9966

Merged

jayzhan211 added 4 commits April 6, 2024 07:42

cleanup

e90464b

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

cleanup

ece925f

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

support roundtrip

89ccc89

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

remmove old format state

41a830a

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

jayzhan211 requested a review from alamb April 6, 2024 02:24

jayzhan211 marked this pull request as draft April 6, 2024 05:42

jayzhan211 marked this pull request as ready for review April 6, 2024 05:43

move aggregate related things to aggr crate

d235d2a

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

jayzhan211 mentioned this pull request Apr 6, 2024

Move conversion of FIRST/LAST Aggregate function to independent physical optimizer rule #9972

Closed

alamb approved these changes Apr 6, 2024

View reviewed changes

jayzhan211 added 5 commits April 6, 2024 20:06

move back to common

51cd272

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

taplo

38b2ce7

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

rm comment

9c7767c

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

cleanup

ea4adde

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

lock

39c5d15

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

jayzhan211 merged commit 2f1c3ab into apache:main Apr 7, 2024

alamb mentioned this pull request Apr 8, 2024

DataFusion weekly project plan (Andrew Lamb) - April 8, 2024 #10002

Closed

9 tasks

waynexia mentioned this pull request May 31, 2024

FIRST/LAST_VALUE behavior changes #10732

Closed

Move First Value UDAF and builtin first / last function to aggregate-functions #9960

Move First Value UDAF and builtin first / last function to aggregate-functions #9960

Uh oh!

Conversation

jayzhan211 commented Apr 5, 2024

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

jayzhan211 commented Apr 5, 2024

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jayzhan211 Apr 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jayzhan211 commented Apr 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jayzhan211 Apr 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jayzhan211 commented Apr 7, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Move First Value UDAF and builtin first / last function to `aggregate-functions` #9960

Move First Value UDAF and builtin first / last function to `aggregate-functions` #9960

jayzhan211 Apr 6, 2024 •

edited

Loading

jayzhan211 commented Apr 6, 2024 •

edited

Loading

jayzhan211 Apr 6, 2024 •

edited

Loading