[feature](nereids) Convert subqueries into algebraic expressions and … #11454

zhengshiJ · 2022-08-03T03:14:04Z

…put them into the query tree

Proposed changes

Issue Number:

Problem summary

1.Convert subqueries to Apply nodes.
2.Convert ApplyNode to ordinary join.

Detailed design:

There are three types of current subexpressions, scalarSubquery, inSubquery, and Exists. The scalarSubquery refers to the returned data as 1 row and 1 column.

Subquery replacement

before:
scalarSubquery:  filter(t1.a = scalarSubquery(output b));
inSubquery:  filter(inSubquery);   inSubquery = (t1.a in select ***);
exists:  filter(exists);   exists = (select ***);

end:
scalarSubquery:  filter(t1.a = b);
inSubquery:  filter(True);
exists:  filter(True);

Subquery Transformation Rules

PushApplyUnderFilter
 * before:
 *             Apply
 *          /              \
 * Input(output:b)    Filter(Correlated predicate/UnCorrelated predicate)
 *
 * after:
 *          Filter(Correlated predicate)
 *                      |
 *                  Apply
 *                /            \
 *      Input(output:b)    Filter(UnCorrelated predicate)

PushApplyUnderProject
 * before:
 *            Apply
 *         /              \
 * Input(output:b)    Project(output:a)
 *
 * after:
 *          Project(b,(if the Subquery is Scalar add 'a' as the output column))
 *          /               \
 * Input(output:b)      Apply

ApplyPullFilterOnAgg
 * before:
 *             Apply
 *          /              \
 * Input(output:b)    agg(output:fn,c; group by:null)
 *                              |
 *              Filter(Correlated predicate(Input.e = this.f)/UnCorrelated predicate)
 *
 * end:
 *          Apply(Correlated predicate(Input.e = this.f))
 *         /              \
 * Input(output:b)    agg(output:fn,this.f; group by:this.f)
 *                              |
 *                    Filter(UnCorrelated predicate)

ApplyPullFilterOnProjectUnderAgg
 * before:
 *              apply
 *         /              \
 * Input(output:b)        agg
 *                         |
 *                  Project(output:a)
 *                         |
 *              Filter(correlated predicate(Input.e = this.f)/Unapply predicate)
 *                          |
 *                         child
 *              apply
 *         /              \
 * Input(output:b)        agg
 *                         |
 *              Filter(correlated predicate(Input.e = this.f)/Unapply predicate)
 *                         |
 *                  Project(output:a,this.f, Unapply predicate(slots))
 *                          |
 *                         child

ScalarToJoin
 * UnCorrelated -> CROSS_JOIN
 * Correlated -> LEFT_OUTER_JOIN

InToJoin
 * Not In -> LEFT_ANTI_JOIN
 * In -> LEFT_SEMI_JOIN

existsToJoin
 * Exists
 *    Correlated -> LEFT_SEMI_JOIN
 *      correlated                  LEFT_SEMI_JOIN(Correlated Predicate)
 *      /       \         -->       /           \
 *    input    queryPlan          input        queryPlan
 *
 *    UnCorrelated -> CROSS_JOIN(limit(1))
 *      uncorrelated                    CROSS_JOIN
 *      /           \          -->      /       \
 *    input        queryPlan          input    limit(1)
 *                                               |
 *                                             queryPlan
 *
 * Not Exists
 *    Correlated -> LEFT_ANTI_JOIN
 *      correlated                  LEFT_ANTI_JOIN(Correlated Predicate)
 *       /       \         -->       /           \
 *     input    queryPlan          input        queryPlan
 *
 *   UnCorrelated -> CROSS_JOIN(Count(*))
 *                                    Filter(count(*) = 0)
 *                                          |
 *         apply                       Cross_Join
 *      /       \         -->       /           \
 *    input    queryPlan          input       agg(output:count(*))
 *                                               |
 *                                             limit(1)
 *                                               |
 *                                             queryPlan

Checklist(Required)

Does it affect the original behavior:
- Yes
- No
- I don't know
Has unit tests been added:
- Yes
- No
- No Need
Has document been added or modified:
- Yes
- No
- No Need
Does it need to update dependencies:
- Yes
- No
Are there any changes that cannot be rolled back:
- Yes (If Yes, please explain WHY)
- No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

924060929 · 2022-08-03T04:04:06Z

fe/fe-core/src/main/java/org/apache/doris/nereids/PlannerContext.java

+        return logicalPlanToSubquery.get(plan);
+    }
+
+    public void setLogicalPlanToSubquery(LogicalPlan plan, SubqueryExpr subqueryExpr) {


Suggested change

public void setLogicalPlanToSubquery(LogicalPlan plan, SubqueryExpr subqueryExpr) {

public void setLogicalPlanToSubqueryIfAbsent(LogicalPlan plan, SubqueryExpr subqueryExpr) {

924060929 · 2022-08-03T04:48:19Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/AnalyzeSubquery.java

+                                            filter.child(), ctx.plannerContext));
+                        })
+                ),
+                RuleType.ANALYZE_AGGREGATE_SUBQUERY.build(


sort and aggregate should not has subqueryExpr? And currently doris not support select subquery.
So you don't need to process LogicalProject, LogicalAggregate and LogicalSort.

924060929 · 2022-08-03T04:49:41Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/AnalyzeSubquery.java

+        );
+    }
+
+    private List<SubqueryExpr> extractSubquery(Expression expression) {


don't need to add extractSubquery function and getAllSubquery function, the shortly function is TreeNode::collect

Suggested change

private List<SubqueryExpr> extractSubquery(Expression expression) {

expression.collect(SubqueryExpr.class::isInstance)

924060929 · 2022-08-03T04:55:52Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/AnalyzeSubquery.java

+    private LogicalPlan addScalarSubqueryCorrelatedJoins(ScalarSubquery scalarSubquery,
+            LogicalPlan childPlan, PlannerContext ctx) {
+        LogicalPlan enforce = new LogicalEnforceSingleRow<>(scalarSubquery.getQueryPlan());
+        scalarSubquery.setAnalyzed(true);


Immutable expression should not has mutable field. if we need some mutable state, we should depend the expression type to compute state, and change it when replace children

e.g.
UnboundExression.isAnalyzed() = false.
other expression.isAnalyzed() = Suppliers.memoized(() -> children().allMatch(Expression::isAnalyzed)).

924060929 · 2022-08-03T06:29:24Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/AnalyzeSubquery.java

+                ctx.getSubquery(exists.getQueryPlan()).getCorrelateSlots());
+    }
+
+    private LogicalPlan appendApplyNode(SubqueryExpr subqueryExpr, LogicalPlan childPlan,


this function not necessary, because the body is only one line and method parameter num is same, and this function is not about 'append'

924060929 · 2022-08-03T06:38:35Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java

-        public Slot visitUnboundSlot(UnboundSlot unboundSlot, Void context) {
+        public Slot visitUnboundSlot(UnboundSlot unboundSlot, PlannerContext context) {
+            List<Slot> tmpBound = bindSlot(unboundSlot, getScope().getSlots());
+            boolean hasCorrelate = false;


this position should has a variable named as boolean foundInThisScope = !tmpBound.isEmpty();
and if foundInThisScope is false, then find in outerScope, if found in outerScope, then set boolean foundInOuterScope = true.

and then is the switch statement, you can declare boolean hasCorrelate = foundInOuterScope or use foundInOuterScope directly

924060929 · 2022-08-03T06:42:22Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java

+            if (tmpBound.size() == 0) {
+                hasCorrelate = true;
+            }
            Optional<List<Slot>> boundedOpt = getScope()


why compute bind slot in thisScope again, I think it should be

if (!foundInThisScope && getScope.getOuterScope().isPresent()) { Optional<List<Slot>> boundedOpt = getScope() .getOuterScope() .get() .toScopeLink() .stream() ... }

924060929 · 2022-08-03T06:47:09Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/Scope.java

+        return checkSlots.contains(slot);
+    }
+
+    public void addSlot(Slot slot) {


checkSlots is final, so can not addSlot

and checkSlots can rename to slotSet

jacktengg · 2022-08-03T08:22:20Z

Please rebase master to re-run P0 test

wangshuo128 · 2022-08-04T03:41:15Z

I suggest first supporting analyzing and unnesting one type of subquery, e.g., converting InSubuqery to joins. Thus we could have a unit test to review the framework and main steps of unnesting subqueries.

924060929 · 2022-08-03T07:47:02Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java

+    /**
+     * Use the visitor to iterate sub expression.
+     */
+    private static class SubExprAnalyzer<C> extends DefaultExpressionRewriter<C> {


Suggested change

private static class SubExprAnalyzer<C> extends DefaultExpressionRewriter<C> {

private static class SubExprAnalyzer extends DefaultExpressionRewriter<PlannerContext> {

924060929 · 2022-09-01T06:44:25Z

fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/ExpressionTranslator.java

+                assertNumRowsElement.getSubqueryString(), translateAsserTion(assertNumRowsElement.getAssertion()));
+    }
+
+    private static org.apache.doris.analysis.AssertNumRowsElement.Assertion translateAsserTion(


Suggested change

private static org.apache.doris.analysis.AssertNumRowsElement.Assertion translateAsserTion(

private static org.apache.doris.analysis.AssertNumRowsElement.Assertion translateAssertion(

924060929 · 2022-09-01T06:46:11Z

fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/ExpressionTranslator.java

+            case GE:
+                return Assertion.GE;
+            default:
+                return null;


return null is safe? should we throw a unsupported type exception here?

924060929 · 2022-09-01T07:07:43Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/AnalyzeSubquery.java

+    }
+
+    /**
+     * Convert expressions with subqueries in filter.


Suggested change

* Convert expressions with subqueries in filter.

* The Subquery in the LogicalFilter will change to LogicalApply, so we must replace the origin Subquery

* in the LogicalFilter(the meaning is remove origin Subquery in the LogicalFilter).

*

* The replace rules are:

924060929 · 2022-09-01T07:25:59Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/AnalyzeSubquery.java

+            return replaceSubquery((SubqueryExpr) oldPredicate);
+        }
+        return newChildren.isEmpty() ? Optional.of(oldPredicate) : Optional.of(oldPredicate.withChildren(
+                newChildren.stream().map(expr -> expr.get()).collect(Collectors.toList())));


.map(expr -> expr.get())

if you not check the Optional whether is present, why not remove Optional? Or you missing filter the present Optional here?

924060929 · 2022-09-01T09:49:48Z

...in/java/org/apache/doris/nereids/rules/rewrite/logical/ApplyPullFilterOnProjectUnderAgg.java

+public class ApplyPullFilterOnProjectUnderAgg extends OneRewriteRuleFactory {
+    @Override
+    public Rule build() {
+        return logicalApply(any(), logicalAggregate(logicalProject(logicalFilter())))


change any() to group()

924060929 · 2022-09-01T09:53:07Z

...in/java/org/apache/doris/nereids/rules/rewrite/logical/ApplyPullFilterOnProjectUnderAgg.java

+                    LogicalAggregate<LogicalProject<LogicalFilter<GroupPlan>>> agg = apply.right();
+                    if (!agg.getGroupByExpressions().isEmpty()) {
+                        return apply;
+                    }


you can move this condition to pattern.

logicalApply( group(), logicalAggregate( logicalProject(logicalFilter())) ).when(agg -> agg.getGroupByExpressions().isEmpty()) ).when(LogicalApply::isCorrelated).then(apply -> {})

924060929 · 2022-09-01T09:59:42Z

...in/java/org/apache/doris/nereids/rules/rewrite/logical/ApplyPullFilterOnProjectUnderAgg.java

+ *                         |
+ *              Filter(correlated predicate(Input.e = this.f)/Unapply predicate)
+ *                         |
+ *                  Project(output:a,child.output)


ditto, this project maybe add some expression that origin filter need, you should note it. And it is not 'child.output'

924060929 · 2022-09-01T10:07:02Z

...core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushApplyUnderProject.java

+public class PushApplyUnderProject extends OneRewriteRuleFactory {
+    @Override
+    public Rule build() {
+        return logicalApply(any(), logicalProject()).when(LogicalApply::isCorrelated).then(apply -> {


change any() to group()

924060929 · 2022-09-01T10:19:02Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/ExistsApplyToJoin.java

+ *     input    queryPlan          input        queryPlan
+ *
+ *    UnCorrelated -> CROSS_JOIN(Count(1))
+ *         apply                Filter(count(1) = 0)


missing CROSS_JOIN?
and Count(1) should change to Count(*)?

and we should add a limit(1) for performance?

filter(equal(cnt, 0)) | aggregate(count(*) as cnt) | limit(1) | queryPlan

count(1) not equal to count(*), maybe wrong.

Forgot to modify the comments here, the code uses count(*), you can add a limit

…put them into the query tree

zhengshiJ force-pushed the subToapp branch from 3328f4f to 860032f Compare August 3, 2022 03:15

github-actions bot added the area/nereids label Aug 3, 2022

924060929 reviewed Aug 3, 2022

View reviewed changes

zhengshiJ force-pushed the subToapp branch from 8f58c4d to 8434d98 Compare August 4, 2022 03:59

zhengshiJ marked this pull request as draft August 9, 2022 07:10

github-actions bot added the area/vectorization label Aug 10, 2022

zhengshiJ force-pushed the subToapp branch from 43db21b to 836133b Compare August 11, 2022 05:37

github-actions bot added the area/planner Issues or PRs related to the query planner label Aug 16, 2022

zhengshiJ force-pushed the subToapp branch from f64dd70 to dfe8fd6 Compare August 17, 2022 03:27

github-actions bot added the kind/test label Aug 18, 2022

zhengshiJ force-pushed the subToapp branch from a76b357 to ef14910 Compare August 18, 2022 06:29

zhengshiJ marked this pull request as ready for review August 19, 2022 07:58

zhengshiJ force-pushed the subToapp branch 5 times, most recently from 47be108 to 89d0bc5 Compare August 31, 2022 02:05

zhengshiJ force-pushed the subToapp branch 3 times, most recently from 9347fc2 to 315945b Compare September 1, 2022 07:12

924060929 reviewed Sep 1, 2022

View reviewed changes

jianghaochen added 7 commits September 2, 2022 12:59

[feature](nereids) Convert subqueries into algebraic expressions and …

f14816e

…put them into the query tree

fix exprid equals

e6f0a8b

fix checkstyle

b1d75b7

fix checkstyle

54c14af

fix pruning

b673726

change subrewritejob

9ff1d0f

fix review

111c1c8

jianghaochen added 2 commits September 2, 2022 12:59

fix test

31572ab

fix test

36eb43f

zhengshiJ force-pushed the subToapp branch from 9ad6250 to 36eb43f Compare September 2, 2022 04:59

jianghaochen added 2 commits September 2, 2022 15:16

fix review

4af9905

fix checkstyle

e5a30d8

924060929 approved these changes Sep 2, 2022

View reviewed changes

924060929 merged commit 7f7a3a7 into apache:master Sep 2, 2022

zhengshiJ mentioned this pull request Sep 5, 2022

[fix](nereids) fix dead loop in unnesting subquery rule #12345

Merged

13 tasks

	public void setLogicalPlanToSubquery(LogicalPlan plan, SubqueryExpr subqueryExpr) {
	public void setLogicalPlanToSubqueryIfAbsent(LogicalPlan plan, SubqueryExpr subqueryExpr) {

	private List<SubqueryExpr> extractSubquery(Expression expression) {
	expression.collect(SubqueryExpr.class::isInstance)

	private static class SubExprAnalyzer<C> extends DefaultExpressionRewriter<C> {
	private static class SubExprAnalyzer extends DefaultExpressionRewriter<PlannerContext> {

	private static org.apache.doris.analysis.AssertNumRowsElement.Assertion translateAsserTion(
	private static org.apache.doris.analysis.AssertNumRowsElement.Assertion translateAssertion(

-     * Convert expressions with subqueries in filter.
+     * The Subquery in the LogicalFilter will change to LogicalApply, so we must replace the origin Subquery
+     * in the LogicalFilter(the meaning is remove origin Subquery in the LogicalFilter).
+     *
+     * The replace rules are:

[feature](nereids) Convert subqueries into algebraic expressions and … #11454

[feature](nereids) Convert subqueries into algebraic expressions and … #11454

Uh oh!

Conversation

zhengshiJ commented Aug 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Problem summary

Detailed design:

Checklist(Required)

Further comments

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

924060929 Aug 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

924060929 Aug 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

924060929 Aug 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jacktengg commented Aug 3, 2022

Uh oh!

wangshuo128 commented Aug 4, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

924060929 Sep 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhengshiJ commented Aug 3, 2022 •

edited

Loading

924060929 Aug 3, 2022 •

edited

Loading

924060929 Aug 3, 2022 •

edited

Loading

924060929 Aug 3, 2022 •

edited

Loading

924060929 Sep 1, 2022 •

edited

Loading