[feature](nereids)support subquery in select list #23271

starocean999 · 2023-08-21T10:07:58Z

Proposed changes

add scalar subquery's output to LogicalApply's output
for in and exists subquery's, add mark join slot into LogicalApply's output
forbid push down alias through join if the project list have any mark join slots.
move normalize aggregate rule to analysis phase

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

starocean999 · 2023-08-21T10:08:07Z

run buildall

starocean999 · 2023-08-24T09:03:38Z

run buildall

hello-stephen · 2023-08-24T09:41:54Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.42 seconds
stream load tsv: 542 seconds loaded 74807831229 Bytes, about 131 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.5 seconds inserted 10000000 Rows, about 338K ops/s
storage size: 17162010894 Bytes

starocean999 · 2023-08-24T13:52:10Z

run buildall

hello-stephen · 2023-08-24T14:21:27Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.94 seconds
stream load tsv: 543 seconds loaded 74807831229 Bytes, about 131 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.6 seconds inserted 10000000 Rows, about 337K ops/s
storage size: 17162084858 Bytes

morrySnow · 2023-08-25T04:01:55Z

fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Analyzer.java

+            topDown(new EliminateGroupByConstant()),
+            topDown(new NormalizeAggregate()),
+            bottomUp(new SubqueryToApply()),
            bottomUp(new CheckAnalysis())


do we still need check again?

morrySnow · 2023-08-25T04:02:51Z

fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java

+                    bottomUp(new PullUpProjectUnderApply()),
+                    topDown(new PushdownFilterThroughProject()),


could we put them into costBased(...) ?

morrySnow · 2023-08-25T04:03:47Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushdownAliasThroughJoin.java

+                (expr instanceof Slot && !(expr instanceof MarkJoinSlotReference))
+                        || (expr instanceof Alias && ((Alias) expr).child() instanceof Slot
+                                && !(((Alias) expr).child() instanceof MarkJoinSlotReference))))


do we need add more ut about this change?

morrySnow · 2023-08-25T04:05:06Z

fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/NormalizeAggregateTest.java

move this file to rules/analyis

morrySnow · 2023-08-25T04:05:09Z

...est/java/org/apache/doris/nereids/rules/rewrite/ExtractAndNormalizeWindowExpressionTest.java

move this file to rules/analyis

morrySnow · 2023-08-25T04:05:12Z

...-core/src/test/java/org/apache/doris/nereids/rules/rewrite/EliminateGroupByConstantTest.java

move this file to rules/analyis

morrySnow · 2023-08-25T04:10:02Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubqueryToApply.java

+                ImmutableList<Set> subqueryExprsList = project.getProjects().stream()
+                        .map(e -> (Set) e.collect(SubqueryExpr.class::isInstance))
+                        .collect(ImmutableList.toImmutableList());


Suggested change

ImmutableList<Set> subqueryExprsList = project.getProjects().stream()

.map(e -> (Set) e.collect(SubqueryExpr.class::isInstance))

.collect(ImmutableList.toImmutableList());

ImmutableList<Set<SubqueryExpr>> subqueryExprsList = project.getProjects().stream()

.<Set<SubqueryExpr>>map(e -> e.collect(SubqueryExpr.class::isInstance))

.collect(ImmutableList.toImmutableList());

morrySnow · 2023-08-25T04:11:23Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubqueryToApply.java

+                if (subqueryExprsList.stream().flatMap(Collection::stream)
+                        .noneMatch(SubqueryExpr.class::isInstance)) {


Suggested change

if (subqueryExprsList.stream().flatMap(Collection::stream)

.noneMatch(SubqueryExpr.class::isInstance)) {

if (subqueryExprsList.stream().flatMap(Collection::stream).count() == 0) {

morrySnow · 2023-08-25T04:12:33Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubqueryToApply.java

+                    return project;
+                }
+                List<NamedExpression> oldProjects = ImmutableList.copyOf(project.getProjects());
+                List<NamedExpression> newProjects = Lists.newArrayList();


use ImmutableList.Builder instead of Lists.newArrayList()

starocean999 · 2023-08-25T09:11:20Z

run buildall

hello-stephen · 2023-08-25T09:40:41Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.14 seconds
stream load tsv: 536 seconds loaded 74807831229 Bytes, about 133 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.5 seconds inserted 10000000 Rows, about 338K ops/s
storage size: 17162070771 Bytes

starocean999 · 2023-08-29T07:52:30Z

run buildall

hello-stephen · 2023-08-29T08:37:45Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.63 seconds
stream load tsv: 550 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162208433 Bytes

github-actions · 2023-08-30T02:21:07Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-08-30T02:21:09Z

PR approved by anyone and no changes requested.

keanji-x · 2023-08-30T03:10:11Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubqueryToApply.java

+                if (subqueryExprsList.stream().flatMap(Collection::stream).count() == 0) {
+                    return project;
+                }
+                List<NamedExpression> oldProjects = ImmutableList.copyOf(project.getProjects());


This copy is redundant?

1. add scalar subquery's output to LogicalApply's output 2. for in and exists subquery's, add mark join slot into LogicalApply's output 3. forbid push down alias through join if the project list have any mark join slots. 4. move normalize aggregate rule to analysis phase

remove float and double literal toString and getStringValue introduced by PR apache#23504 and PR apache#23271 These functions lead to wrong cast result of double and float literal

remove float and double literal toString and getStringValue introduced by PR apache#23504 and PR apache#23271 These functions lead to wrong cast result of double and float literal fix string cast to datetimev2 error introduced by PR apache#26827 we should cast to exactly scale of datetimev2 when cast string to it

…ed (#28959) FIX 1. remove float and double literal toString and getStringValue introduced by PR #23504 and PR #23271 These functions lead to wrong cast result of double and float literal 2. fix compute signature for datetimev2 always produce scale 6 3. fix stats calculator failed when generate node stats with two same column 4. constant fold on fe failed when cast double to integral TODO after fix the first problem, some mv matching not work well, fix them later - test_dup_mv_div - test_dup_mv_json - test_tcu

…ed (apache#28959) FIX 1. remove float and double literal toString and getStringValue introduced by PR apache#23504 and PR apache#23271 These functions lead to wrong cast result of double and float literal 2. fix compute signature for datetimev2 always produce scale 6 3. fix stats calculator failed when generate node stats with two same column 4. constant fold on fe failed when cast double to integral TODO after fix the first problem, some mv matching not work well, fix them later - test_dup_mv_div - test_dup_mv_json - test_tcu

starocean999 force-pushed the master_bug_1222 branch from b051155 to 505c9ab Compare August 24, 2023 08:53

morrySnow reviewed Aug 25, 2023

View reviewed changes

starocean999 force-pushed the master_bug_1222 branch from f453c72 to ade7674 Compare August 25, 2023 09:11

starocean999 added 4 commits August 29, 2023 15:46

[feature](nereids)support subquery in select list

63e742c

fix failed test cases

ae81dc3

fix fe ut

ee80123

modified based on comments

cd18d91

starocean999 force-pushed the master_bug_1222 branch from ade7674 to cd18d91 Compare August 29, 2023 07:51

morrySnow approved these changes Aug 30, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 30, 2023

github-actions bot added the reviewed label Aug 30, 2023

morrySnow added the dev/2.0.2 label Aug 30, 2023

keanji-x approved these changes Aug 30, 2023

View reviewed changes

morrySnow merged commit 7379cdc into apache:master Aug 31, 2023

xiaokang added merge_conflict dev/2.0.2-merged and removed dev/2.0.2 labels Aug 31, 2023

morrySnow mentioned this pull request Dec 28, 2023

[fix](Nereids) create double literal when create decimal literal failed #28959

Merged

		bottomUp(new PullUpProjectUnderApply()),
		topDown(new PushdownFilterThroughProject()),

		if (subqueryExprsList.stream().flatMap(Collection::stream)
		.noneMatch(SubqueryExpr.class::isInstance)) {

[feature](nereids)support subquery in select list #23271

[feature](nereids)support subquery in select list #23271

Uh oh!

Conversation

starocean999 commented Aug 21, 2023 • edited by morrySnow Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

starocean999 commented Aug 21, 2023

Uh oh!

starocean999 commented Aug 24, 2023

Uh oh!

hello-stephen commented Aug 24, 2023

Uh oh!

starocean999 commented Aug 24, 2023

Uh oh!

hello-stephen commented Aug 24, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

starocean999 commented Aug 25, 2023

Uh oh!

hello-stephen commented Aug 25, 2023

Uh oh!

starocean999 commented Aug 29, 2023

Uh oh!

hello-stephen commented Aug 29, 2023

Uh oh!

github-actions bot commented Aug 30, 2023

Uh oh!

github-actions bot commented Aug 30, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

starocean999 commented Aug 21, 2023 •

edited by morrySnow

Loading