[feature](statistics) Statistics derivation.Step 1:ScanNode implement… #8947

zhengshiJ · 2022-04-11T02:09:59Z

The purpose of this function is to modify the derivation of the current statistics.
There are two modifications:

The statistical information derivation is removed from the Node node.
Modify the current way of obtaining cardinality, and change it to obtain it from statistical information.
For details on the modification method, see: https://shimo.im/docs/473QyOzWBjHMlZ3w
The implementation of this function is divided into multiple submissions, and this submission implements the derivation of scan_node statistics.

Proposed changes

Issue Number: close #9241

Problem Summary:

The derivation of the current statistical information is coupled in the node node, and the method of calculating the cardinality is carried out by the mock method, and the ndv value is not obtained correctly.

Checklist(Required)

Does it affect the original behavior: (No)
Has unit tests been added: (No)
Has document been added or modified: (No Need)
Does it need to update dependencies: (No)
Are there any changes that cannot be rolled back: (No)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java

fe/fe-core/src/main/java/org/apache/doris/statistics/StatsDeriveResult.java

fe/fe-core/src/main/java/org/apache/doris/statistics/BaseStatsDerive.java

fe/fe-core/src/main/java/org/apache/doris/statistics/StatsRecursiveDerive.java

fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java

fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java

fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java

fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java

fe/fe-core/src/main/java/org/apache/doris/planner/ScanNode.java

fe/fe-core/src/main/java/org/apache/doris/planner/BrokerScanNode.java

fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java

fe/fe-core/src/main/java/org/apache/doris/statistics/BaseStatsDerive.java

fe/fe-core/src/main/java/org/apache/doris/statistics/ScanStatsDerive.java

fe/fe-core/src/main/java/org/apache/doris/statistics/Statistics.java

fe/fe-core/src/main/java/org/apache/doris/statistics/StatsRecursiveDerive.java

fe/fe-core/src/main/java/org/apache/doris/statistics/StatsDeriveResult.java

fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java

fe/fe-core/src/main/java/org/apache/doris/statistics/OlapScanStatsDerive.java

fe/fe-core/src/main/java/org/apache/doris/statistics/StatsDeriveResult.java

fe/fe-core/src/main/java/org/apache/doris/statistics/BaseStatsDerive.java

EmmyMiao87 · 2022-04-22T08:46:33Z

fe/fe-core/src/main/java/org/apache/doris/statistics/BaseStatsDerive.java

+
+    // Currently it simply adds the number of rows of children
+    protected long deriveRowCount() {
+        applyConjunctsSelectivity();


Suggested change

applyConjunctsSelectivity();

rowcount = children max(rowcount)

applyConjunctsSelectivity();

fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java

fe/fe-core/src/main/java/org/apache/doris/statistics/BaseStatsDerive.java

EmmyMiao87

LGTM

github-actions · 2022-04-25T04:44:05Z

PR approved by at least one committer and no changes requested.

github-actions · 2022-04-25T04:44:07Z

PR approved by anyone and no changes requested.

EmmyMiao87 · 2022-04-25T04:45:51Z

The FE ut is failed case:

AlterViewStmtTest.testNormal
TableQueryPlanActionTest.testQueryPlanAction

Those case will be fixed in PR: #9158

The BE unit test problem is independent of the current pr.

EmmyMiao87

LGTM

github-actions · 2022-04-26T10:38:27Z

PR approved by at least one committer and no changes requested.

EmmyMiao87

The code base line is incorrect

…ation

…blem

EmmyMiao87

LGTM

github-actions · 2022-04-28T02:51:52Z

PR approved by at least one committer and no changes requested.

apache#8947) * [feature](statistics) Statistics derivation.Step 1:ScanNode implementation Co-authored-by: jianghaochen <jianghaochen@meituan.com>

github-actions bot added the area/planner Issues or PRs related to the query planner label Apr 11, 2022

zhengshiJ force-pushed the master branch from be95006 to 9c0061e Compare April 11, 2022 02:46

zbtzbtzbt reviewed Apr 11, 2022

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java Outdated Show resolved Hide resolved

EmmyMiao87 added the area/statistics Issues or PRS related to statistics label Apr 13, 2022

EmmyMiao87 self-assigned this Apr 13, 2022

EmmyMiao87 reviewed Apr 14, 2022

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/statistics/StatsRecursiveDerive.java Outdated Show resolved Hide resolved

fe/fe-core/src/main/java/org/apache/doris/statistics/StatsRecursiveDerive.java Outdated Show resolved Hide resolved

EmmyMiao87 reviewed Apr 14, 2022

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java Show resolved Hide resolved

zhengshiJ force-pushed the master branch from 46e8892 to 8d231ff Compare April 15, 2022 04:29

EmmyMiao87 reviewed Apr 20, 2022

View reviewed changes

zhengshiJ force-pushed the master branch 2 times, most recently from 66972cf to 9c2ba3a Compare April 20, 2022 06:38

github-actions bot added the area/load Issues or PRs related to all kinds of load label Apr 20, 2022

EmmyMiao87 reviewed Apr 20, 2022

View reviewed changes

zhengshiJ force-pushed the master branch from 24fb1e5 to 9d7cd85 Compare April 21, 2022 06:49

EmmyMiao87 reviewed Apr 22, 2022

View reviewed changes

zhengshiJ force-pushed the master branch from 4f7a5e7 to 26f6032 Compare April 22, 2022 10:39

EmmyMiao87 reviewed Apr 22, 2022

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/statistics/BaseStatsDerive.java Show resolved Hide resolved

zhengshiJ force-pushed the master branch from 26f6032 to 789ab67 Compare April 24, 2022 02:06

EmmyMiao87 previously approved these changes Apr 25, 2022

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 25, 2022

github-actions bot added the reviewed label Apr 25, 2022

zhengshiJ dismissed EmmyMiao87’s stale review via 6f8b0dc April 26, 2022 02:32

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 26, 2022

EmmyMiao87 approved these changes Apr 26, 2022

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 26, 2022

EmmyMiao87 requested changes Apr 26, 2022

View reviewed changes

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 26, 2022

zhengshiJ force-pushed the master branch from 4735652 to f56fcda Compare April 27, 2022 03:42

github-actions bot added area/routine load area/spark-load Issues or PRs related to the spark load area/sql/function Issues or PRs related to the SQL functions area/vectorization kind/docs Categorizes issue or PR as related to documentation. kind/test labels Apr 27, 2022

zhengshiJ force-pushed the master branch from f56fcda to 789ab67 Compare April 27, 2022 03:56

jianghaochen added 3 commits April 27, 2022 12:12

[feature](statistics) Statistics derivation.Step 1:ScanNode implement…

ee85f1c

…ation

Fix problems in review

9fbbece

Modify the review question

1e4da1c

zhengshiJ force-pushed the master branch from 789ab67 to fb4927e Compare April 27, 2022 04:14

zhengshiJ mentioned this pull request Apr 27, 2022

[feature](statistics) Statistics derivation.Step 1:ScanNode implement… #9258

Closed

Fix the ut problem introduced by apache#9047 and fixed the review pro…

25bd407

…blem

zhengshiJ force-pushed the master branch from 95d5d31 to 25bd407 Compare April 27, 2022 12:47

EmmyMiao87 approved these changes Apr 28, 2022

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 28, 2022

yiguolei merged commit 9ef09b8 into apache:master Apr 29, 2022

morningman added the dev/backlog waiting to be merged in future dev branch label Apr 29, 2022

	applyConjunctsSelectivity();
	rowcount = children max(rowcount)
	applyConjunctsSelectivity();

[feature](statistics) Statistics derivation.Step 1:ScanNode implement… #8947

[feature](statistics) Statistics derivation.Step 1:ScanNode implement… #8947

Uh oh!

Conversation

zhengshiJ commented Apr 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Problem Summary:

Checklist(Required)

Further comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

EmmyMiao87 Apr 22, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

EmmyMiao87 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 25, 2022

Uh oh!

github-actions bot commented Apr 25, 2022

Uh oh!

EmmyMiao87 commented Apr 25, 2022

Uh oh!

EmmyMiao87 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 26, 2022

Uh oh!

EmmyMiao87 left a comment

Choose a reason for hiding this comment

Uh oh!

EmmyMiao87 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 28, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

zhengshiJ commented Apr 11, 2022 •

edited

Loading