[feature](nereids)support window function #14397

YangShaw · 2022-11-18T11:05:18Z

Proposed changes

Issue Number: close #xxx

Problem summary

support WindowFunction in nereids;

Checklist(Required)

Does it affect the original behavior:
- Yes
- No
- I don't know
Has unit tests been added:
- Yes
- No
- No Need
Has document been added or modified:
- Yes
- No
- No Need
Does it need to update dependencies:
- Yes
- No
Are there any changes that cannot be rolled back:
- Yes (If Yes, please explain WHY)
- No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

hello-stephen · 2022-12-28T13:38:24Z

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.21 seconds
stream load tsv: 462 seconds loaded 74807831229 Bytes, about 154 MB/s
stream load json: 39 seconds loaded 2358488459 Bytes, about 57 MB/s
stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230213131555_clickbench_pr_95395.html

fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4

fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java

fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalWindow.java

morrySnow · 2023-02-01T07:08:32Z

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/algebra/Window.java

+    /**
+     * translate WindowFrame to AnalyticWindow
+     */
+    default AnalyticWindow translateWindowFrame(WindowFrame windowFrame, PlanTranslatorContext context) {
+        FrameUnitsType frameUnits = windowFrame.getFrameUnits();
+        FrameBoundary leftBoundary = windowFrame.getLeftBoundary();
+        FrameBoundary rightBoundary = windowFrame.getRightBoundary();
+
+        AnalyticWindow.Type type = frameUnits == FrameUnitsType.ROWS
+                ? AnalyticWindow.Type.ROWS : AnalyticWindow.Type.RANGE;
+
+        AnalyticWindow.Boundary left = withFrameBoundary(leftBoundary, context);
+        AnalyticWindow.Boundary right = withFrameBoundary(rightBoundary, context);
+
+        return new AnalyticWindow(type, left, right);
+    }


i think it it better that put translate functions in translator

due to that AnalyticWindow doesn't extend Expr, it is hard to use visitWindowFrame() in ExpressionTranslator. And in consider of that the code related of translate windowFrame is long, I put these functions in Window interface rather than PhysicalPlanTranslator.

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Window.java

fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java

...rc/main/java/org/apache/doris/nereids/trees/expressions/functions/window/FrameUnitsType.java

morrySnow · 2023-02-01T08:01:17Z

fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java

+                    if (containsWindowExpressions(projects)) {
+                        return new LogicalWindow<>(projects, input);
+                    }
+                    return new LogicalProject<>(projects, Collections.emptyList(), input, isDistinct);


maybe we should generate LogicalWindow after we do bind.
we chould just treat windowExpression as a scalar expression.

if we use window function without agg. Then window function will in Projections.
we could generate LogicalProject(LogicalWindow(LogicalProject)) to handle all case.

if we use window function in agg. Then window function will in Agg's output expressions. After normalize agg it will present in the Project topping on Aggregate. and then we could do the same thing as first scene.

...c/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalWindow.java

...c/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java

...rc/main/java/org/apache/doris/nereids/trees/expressions/functions/window/WindowFunction.java

.../main/java/org/apache/doris/nereids/trees/expressions/functions/window/FirstOrLastValue.java

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/window/Lag.java

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/window/Lead.java

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/window/Ntile.java

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalQuickSort.java

fe/fe-core/src/test/java/org/apache/doris/nereids/parser/NereidsParserTest.java

fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java

fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java

morrySnow · 2023-02-13T10:11:50Z

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/WindowExpression.java

+ * which is an UnboundFunction at first and will be analyzed as relevant BoundFunction
+ * (can be a WindowFunction or AggregateFunction) after BindFunction.
+ */
+public class WindowExpression extends Expression implements PropagateNullable {


Suggested change

public class WindowExpression extends Expression implements PropagateNullable {

public class WindowExpression extends Expression {

@Override

public boolean nullable() {

return function.nullable();

}

morrySnow · 2023-02-13T10:12:08Z

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/window/Ntile.java

+    @Override
+    public DataType getDataType() {
+        return IntegerType.INSTANCE;
+    }


remove this function

morrySnow · 2023-02-13T10:15:00Z

...ore/src/main/java/org/apache/doris/nereids/trees/expressions/functions/window/DenseRank.java

+    @Override
+    public DataType getDataType() {
+        return IntegerType.INSTANCE;
+    }


remove this

morrySnow · 2023-02-13T10:17:46Z

regression-test/suites/nereids_syntax_p0/window_function.groovy

+        SELECT *, row_number() over(partition by c1)
+        FROM (
+            SELECT *, row_number() over(partition by c2)
+            FROM window_test


need order by in over to get a stable result

morrySnow · 2023-02-13T10:36:59Z

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/window/Ntile.java

+            FunctionSignature.ret(LargeIntType.INSTANCE).args(LargeIntType.INSTANCE)
+    );
+
+    private Expression buckets;


bucket is not use anymore

github-actions · 2023-02-13T13:02:35Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-02-13T13:02:40Z

PR approved by anyone and no changes requested.

fix distinct window compute wrong result, introduced by apache#14397 ```sql select distinct sum(value) over(partition by id) from ( select 100 value, 1 id union all select 100, 2 )a; +----------------------------------+ | sum(value) over(partition by id) | +----------------------------------+ | 100 | | 100 | +----------------------------------+ ```

### What problem does this PR solve? Related PR: #21727 #14397 Problem Summary: 1. forgot to copy isChecked flag in LogicalWindow when do deep copy 2. implement LogicalWindow To PhyscialWindow should not check isChecked flag This PR: 1. check deep copy for all plan node 2. remove check isChecked in LogicalWindow To PhyscialWindow

Related PR: #21727 #14397 Problem Summary: 1. forgot to copy isChecked flag in LogicalWindow when do deep copy 2. implement LogicalWindow To PhyscialWindow should not check isChecked flag This PR: 1. check deep copy for all plan node 2. remove check isChecked in LogicalWindow To PhyscialWindow

fix distinct window compute wrong result, introduced by apache#14397 ```sql select distinct sum(value) over(partition by id) from ( select 100 value, 1 id union all select 100, 2 )a; +----------------------------------+ | sum(value) over(partition by id) | +----------------------------------+ | 100 | | 100 | +----------------------------------+ ```

### What problem does this PR solve? Related PR: apache#21727 apache#14397 Problem Summary: 1. forgot to copy isChecked flag in LogicalWindow when do deep copy 2. implement LogicalWindow To PhyscialWindow should not check isChecked flag This PR: 1. check deep copy for all plan node 2. remove check isChecked in LogicalWindow To PhyscialWindow

github-actions bot added area/nereids area/planner Issues or PRs related to the query planner labels Nov 18, 2022

YangShaw force-pushed the analytic_node branch 3 times, most recently from 2deb44f to 04696e9 Compare November 21, 2022 17:20

YangShaw force-pushed the analytic_node branch from f6b2091 to 4a05b19 Compare December 28, 2022 08:11

YangShaw force-pushed the analytic_node branch from f8de731 to 8ba1a4f Compare January 16, 2023 10:55

YangShaw marked this pull request as ready for review January 18, 2023 05:49

github-actions bot added the kind/test label Jan 18, 2023

YangShaw force-pushed the analytic_node branch from 3c4e44a to 06b2082 Compare January 18, 2023 11:41

YangShaw force-pushed the analytic_node branch 3 times, most recently from 09c9cda to f81180b Compare January 31, 2023 03:33

morrySnow reviewed Feb 1, 2023

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java Outdated Show resolved Hide resolved

YangShaw force-pushed the analytic_node branch 2 times, most recently from 780f3c9 to 3c3cd0f Compare February 10, 2023 01:35

fuyu29 added 8 commits February 10, 2023 16:46

window function

3f03c66

rebase

a2a8bc0

temp change pull_request_id

dac1046

fix window functions

277545e

fix window(agg)

1f9afbb

rebase

7da5744

fix

caa22e1

fix reg

81fb58d

YangShaw force-pushed the analytic_node branch from feb9aad to 81fb58d Compare February 10, 2023 09:07

fuyu29 added 2 commits February 11, 2023 13:23

fix

ffcda18

fix

423948a

morrySnow reviewed Feb 13, 2023

View reviewed changes

fuyu29 added 5 commits February 13, 2023 18:43

fix ntile & dense_rank

135f39d

fix reg

4ae669f

fix

9d496a9

fix

90a5cec

fix

809230d

morrySnow approved these changes Feb 13, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 13, 2023

github-actions bot added the reviewed label Feb 13, 2023

morrySnow merged commit 77a3288 into apache:master Feb 13, 2023

YangShaw added a commit to YangShaw/doris that referenced this pull request Feb 17, 2023

[feature](Nereids) support window function (apache#14397)

ba14e20

morrySnow mentioned this pull request Mar 10, 2025

[fix](Nereids) deep copy for LogicalWindow is wrong #48861

Merged

16 tasks

924060929 mentioned this pull request Mar 12, 2025

[fix](nereids) fix distinct window compute wrong result #48987

Closed

16 tasks

-public class WindowExpression extends Expression implements PropagateNullable {
+public class WindowExpression extends Expression {
+    @Override
+    public boolean nullable() {
+        return function.nullable();
+    }

[feature](nereids)support window function #14397

[feature](nereids)support window function #14397

Uh oh!

Conversation

YangShaw commented Nov 18, 2022 • edited by morrySnow Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Problem summary

Checklist(Required)

Further comments

Uh oh!

hello-stephen commented Dec 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 13, 2023

Uh oh!

github-actions bot commented Feb 13, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

YangShaw commented Nov 18, 2022 •

edited by morrySnow

Loading

hello-stephen commented Dec 28, 2022 •

edited

Loading