planner: add aggregation hints `TIDB_HASHAGG` and `TIDB_STREAMAGG` by foreyes · Pull Request #11364 · pingcap/tidb

foreyes · 2019-07-22T06:28:59Z

What problem does this PR solve?

Add Optimizer Hints TIDB_HASHAGG and TIDB_STREAMAGG.

What is changed and how it works?

Handle the hint from parser, and enforce planner to choose the aggregation type.
Related parser PR: pingcap/parser#394

mysql> explain select count(*) from t t1, t t2 where t1.a = t2.b;
+--------------------------+----------+------+--------------------------------------------------------------------+
| id                       | count    | task | operator info                                                      |
+--------------------------+----------+------+--------------------------------------------------------------------+
| StreamAgg_13             | 1.00     | root | funcs:count(1)                                                     |
| └─HashLeftJoin_26        | 12500.00 | root | inner join, inner:TableReader_20, equal:[eq(test.t1.a, test.t2.b)] |
|   ├─TableReader_22       | 10000.00 | root | data:TableScan_21                                                  |
|   │ └─TableScan_21       | 10000.00 | cop  | table:t1, range:[-inf,+inf], keep order:false, stats:pseudo        |
|   └─TableReader_20       | 10000.00 | root | data:TableScan_19                                                  |
|     └─TableScan_19       | 10000.00 | cop  | table:t2, range:[-inf,+inf], keep order:false, stats:pseudo        |
+--------------------------+----------+------+--------------------------------------------------------------------+
6 rows in set (0.00 sec)

mysql> explain select /*+ TIDB_HASHAGG() */ count(*) from t t1, t t2 where t1.a = t2.b;
+--------------------------+----------+------+--------------------------------------------------------------------+
| id                       | count    | task | operator info                                                      |
+--------------------------+----------+------+--------------------------------------------------------------------+
| HashAgg_11               | 1.00     | root | funcs:count(1)                                                     |
| └─HashLeftJoin_15        | 12500.00 | root | inner join, inner:TableReader_18, equal:[eq(test.t1.a, test.t2.b)] |
|   ├─TableReader_20       | 10000.00 | root | data:TableScan_19                                                  |
|   │ └─TableScan_19       | 10000.00 | cop  | table:t1, range:[-inf,+inf], keep order:false, stats:pseudo        |
|   └─TableReader_18       | 10000.00 | root | data:TableScan_17                                                  |
|     └─TableScan_17       | 10000.00 | cop  | table:t2, range:[-inf,+inf], keep order:false, stats:pseudo        |
+--------------------------+----------+------+--------------------------------------------------------------------+
6 rows in set (0.01 sec)

mysql> explain select count(t1.a) from t t1, t t2 where t1.a = t2.a*2 group by t1.a;
+--------------------------+----------+------+---------------------------------------------------------------------------+
| id                       | count    | task | operator info                                                             |
+--------------------------+----------+------+---------------------------------------------------------------------------+
| HashAgg_13               | 8000.00  | root | group by:test.t1.a, funcs:count(test.t1.a)                                |
| └─HashLeftJoin_16        | 12500.00 | root | inner join, inner:Projection_21, equal:[eq(test.t1.a, mul(test.t2.a, 2))] |
|   ├─TableReader_20       | 10000.00 | root | data:TableScan_19                                                         |
|   │ └─TableScan_19       | 10000.00 | cop  | table:t1, range:[-inf,+inf], keep order:false, stats:pseudo               |
|   └─Projection_21        | 10000.00 | root | test.t2.a, mul(test.t2.a, 2)                                              |
|     └─TableReader_23     | 10000.00 | root | data:TableScan_22                                                         |
|       └─TableScan_22     | 10000.00 | cop  | table:t2, range:[-inf,+inf], keep order:false, stats:pseudo               |
+--------------------------+----------+------+---------------------------------------------------------------------------+
7 rows in set (0.00 sec)

mysql> explain select /*+ TIDB_STREAMAGG() */ count(t1.a) from t t1, t t2 where t1.a = t2.a*2 group by t1.a;
+----------------------------+----------+------+---------------------------------------------------------------------------+
| id                         | count    | task | operator info                                                             |
+----------------------------+----------+------+---------------------------------------------------------------------------+
| StreamAgg_15               | 8000.00  | root | group by:test.t1.a, funcs:count(test.t1.a)                                |
| └─Sort_24                  | 12500.00 | root | test.t1.a:asc                                                             |
|   └─HashLeftJoin_16        | 12500.00 | root | inner join, inner:Projection_21, equal:[eq(test.t1.a, mul(test.t2.a, 2))] |
|     ├─TableReader_20       | 10000.00 | root | data:TableScan_19                                                         |
|     │ └─TableScan_19       | 10000.00 | cop  | table:t1, range:[-inf,+inf], keep order:false, stats:pseudo               |
|     └─Projection_21        | 10000.00 | root | test.t2.a, mul(test.t2.a, 2)                                              |
|       └─TableReader_23     | 10000.00 | root | data:TableScan_22                                                         |
|         └─TableScan_22     | 10000.00 | cop  | table:t2, range:[-inf,+inf], keep order:false, stats:pseudo               |
+----------------------------+----------+------+---------------------------------------------------------------------------+
8 rows in set (0.00 sec)

Check List

Tests

Unit test

Code changes

Change plan builder to handling aggregation hints.
Change exhaust physical plan to apply aggregation hints.

Side effects

Change optimizer behaviors.

Related changes

Add new rule in parser

codecov · 2019-07-22T07:38:30Z

Codecov Report

Merging #11364 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #11364   +/-   ##
===========================================
  Coverage   81.6243%   81.6243%           
===========================================
  Files           426        426           
  Lines         93640      93640           
===========================================
  Hits          76433      76433           
  Misses        11807      11807           
  Partials       5400       5400

foreyes · 2019-07-24T06:03:33Z

/run-all-tests

foreyes · 2019-07-24T06:20:39Z

PTAL. @zz-jason @lamxTyler

foreyes · 2019-07-24T06:57:54Z

/run-all-tests

foreyes · 2019-07-24T09:56:14Z

Code improved, PTAL. @zz-jason @XuHuaiyu

XuHuaiyu

LGTM

XuHuaiyu · 2019-07-25T02:36:16Z

We should update the version of parser in go.mod before merging this commit.

foreyes · 2019-08-06T04:32:43Z

PTAL. @XuHuaiyu @eurekaka

XuHuaiyu · 2019-08-06T05:29:02Z

+
 	all, desc := prop.AllSameOrder()
-	if len(la.possibleProperties) == 0 || !all {
+	if !all {


This is not resolved?

XuHuaiyu · 2019-08-06T05:31:54Z

 	}
 }
+
+func (s *testPlanSuite) TestAggregationHints(c *C) {


Add a test case which contains subquery?

For the first one, because possibleChildProperties are only possible... We'd better not rely too much on it, I handle this in line 1272 - 1275, you can take a look.

For the test case, I will add them soon.

When adding test case, find another bug. Fix it soon...

foreyes · 2019-08-06T07:50:39Z

Add tests and fix a Merge Join bug, PTAL. @eurekaka @XuHuaiyu

eurekaka

LGTM

XuHuaiyu

LGTM

foreyes · 2019-08-07T02:45:51Z

/run-all-tests

XuHuaiyu · 2019-08-07T02:50:18Z

I'm still curious that, is this expectable?
Or, this will be fixed in another PR?

CREATE TABLE `t` (
  `a` int(11) DEFAULT NULL,
  `b` int(11) DEFAULT NULL,
  KEY `a` (`a`)
);
tidb> desc select /*+ TIDB_HJ(t)  */ a, count(b) from t group by a order by a;
+--------------------------+----------+------+--------------------------------------------------------------------+
| id                       | count    | task | operator info                                                      |
+--------------------------+----------+------+--------------------------------------------------------------------+
| Projection_22            | 8000.00  | root | test.t.a, 2_col_0                                                  |
| └─StreamAgg_24           | 8000.00  | root | group by:test.t.a, funcs:count(test.t.b), firstrow(test.t.a)       |
|   └─Projection_21        | 10000.00 | root | test.t.a, test.t.b                                                 |
|     └─IndexLookUp_20     | 10000.00 | root |                                                                    |
|       ├─IndexScan_18     | 10000.00 | cop  | table:t, index:a, range:[NULL,+inf], keep order:true, stats:pseudo |
|       └─TableScan_19     | 10000.00 | cop  | table:t, keep order:false, stats:pseudo                            |
+--------------------------+----------+------+--------------------------------------------------------------------+
6 rows in set, 1 warning (0.00 sec)

tidb> show warnings;
+---------+------+-----------------------------------------------------------------------------------------------------------------------+
| Level   | Code | Message                                                                                                               |
+---------+------+-----------------------------------------------------------------------------------------------------------------------+
| Warning | 1815 | There are no matching table names for (t) in optimizer hint /*+ TIDB_HJ(t) */. Maybe you can use the table alias name |
+---------+------+-----------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

foreyes · 2019-08-07T02:53:07Z

I'm still curious that, is this expectable?
Or, this will be fixed in another PR?

CREATE TABLE `t` (
  `a` int(11) DEFAULT NULL,
  `b` int(11) DEFAULT NULL,
  KEY `a` (`a`)
);
tidb> desc select /*+ TIDB_HJ(t)  */ a, count(b) from t group by a order by a;
+--------------------------+----------+------+--------------------------------------------------------------------+
| id                       | count    | task | operator info                                                      |
+--------------------------+----------+------+--------------------------------------------------------------------+
| Projection_22            | 8000.00  | root | test.t.a, 2_col_0                                                  |
| └─StreamAgg_24           | 8000.00  | root | group by:test.t.a, funcs:count(test.t.b), firstrow(test.t.a)       |
|   └─Projection_21        | 10000.00 | root | test.t.a, test.t.b                                                 |
|     └─IndexLookUp_20     | 10000.00 | root |                                                                    |
|       ├─IndexScan_18     | 10000.00 | cop  | table:t, index:a, range:[NULL,+inf], keep order:true, stats:pseudo |
|       └─TableScan_19     | 10000.00 | cop  | table:t, keep order:false, stats:pseudo                            |
+--------------------------+----------+------+--------------------------------------------------------------------+
6 rows in set, 1 warning (0.00 sec)

tidb> show warnings;
+---------+------+-----------------------------------------------------------------------------------------------------------------------+
| Level   | Code | Message                                                                                                               |
+---------+------+-----------------------------------------------------------------------------------------------------------------------+
| Warning | 1815 | There are no matching table names for (t) in optimizer hint /*+ TIDB_HJ(t) */. Maybe you can use the table alias name |
+---------+------+-----------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

I know this case, it's expected, but looks weird, I will fix it in another PR. @XuHuaiyu

…ingcap#11364)

foreyes added sig/planner SIG: Planner status/WIP type/new-feature labels Jul 22, 2019

foreyes mentioned this pull request Jul 22, 2019

parser: add TableOptimizerHint TIDB_HASHAGG and TIDB_STREAMAGG pingcap/parser#394

Merged

foreyes force-pushed the dev/add_agg_hints branch 2 times, most recently from ecd3e00 to ff1f239 Compare July 23, 2019 07:05

foreyes removed the status/WIP label Jul 24, 2019

foreyes requested review from alivxxx and zz-jason July 24, 2019 06:19

foreyes changed the title ~~[WIP] planner: add aggregation hints TIDB_HASHAGG and TIDB_STREAMAGG~~ planner: add aggregation hints TIDB_HASHAGG and TIDB_STREAMAGG Jul 24, 2019

alivxxx reviewed Jul 24, 2019

View reviewed changes

Comment thread planner/core/exhaust_physical_plans.go Outdated

Comment thread planner/core/logical_plans.go Outdated

foreyes force-pushed the dev/add_agg_hints branch from 37eff4e to 5accbd8 Compare July 24, 2019 06:53

XuHuaiyu reviewed Jul 24, 2019

View reviewed changes

Comment thread planner/core/logical_plans.go Outdated

Comment thread planner/core/logical_plan_builder.go Outdated

Comment thread planner/core/exhaust_physical_plans.go Outdated

zz-jason removed their request for review July 24, 2019 08:39

foreyes force-pushed the dev/add_agg_hints branch from fa37100 to 2c46284 Compare July 24, 2019 09:42

XuHuaiyu reviewed Jul 25, 2019

View reviewed changes

XuHuaiyu added the status/LGT1 Indicates that a PR has LGTM 1. label Jul 25, 2019

foreyes force-pushed the dev/add_agg_hints branch 3 times, most recently from 80af97d to 74c1bf3 Compare July 26, 2019 09:52

foreyes added 4 commits July 30, 2019 10:15

add aggregation hints

e87fad5

try to fix ci

0184392

add tests

1f0e8c4

add enforce flag to stream agg hint & fix test

9a1a791

foreyes added 2 commits August 5, 2019 20:01

Merge remote-tracking branch 'upstream/master' into dev/add_agg_hints

0b1acff

resolve comments

7dd61f9

alivxxx reviewed Aug 6, 2019

View reviewed changes

Comment thread executor/join_test.go Outdated

avoid test leak

d2246e5

foreyes mentioned this pull request Aug 6, 2019

Extract a common error message pattern for conflicted SQL hints and hint alias #11632

Open

foreyes requested review from XuHuaiyu and eurekaka August 6, 2019 04:32

XuHuaiyu reviewed Aug 6, 2019

View reviewed changes

foreyes added 3 commits August 6, 2019 15:10

add tests & fix bug

0a122d6

fix explain test

8c49116

fix test

9d4c316

eurekaka reviewed Aug 6, 2019

View reviewed changes

Merge branch 'master' into dev/add_agg_hints

2609f8b

eurekaka added the status/LGT1 Indicates that a PR has LGTM 1. label Aug 6, 2019

XuHuaiyu approved these changes Aug 7, 2019

View reviewed changes

Merge branch 'master' into dev/add_agg_hints

081b602

XuHuaiyu added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Aug 7, 2019

foreyes merged commit a530f87 into pingcap:master Aug 7, 2019

foreyes deleted the dev/add_agg_hints branch August 7, 2019 02:51

foreyes mentioned this pull request Aug 14, 2019

planner: revert optimizer hint scope #11740

Merged

foreyes added a commit to foreyes/tidb that referenced this pull request Sep 16, 2019

planner: add aggregation hints TIDB_HASHAGG and TIDB_STREAMAGG (p…

2dd2878

…ingcap#11364)

foreyes mentioned this pull request Sep 16, 2019

planner: cherry-pick sql hint related PRs into release-3.1 #12192

Merged

zz-jason mentioned this pull request Oct 10, 2019

Feature Request: Richer SQL Hints #12304

Closed

Conversation

foreyes commented Jul 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Check List

Uh oh!

codecov Bot commented Jul 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

foreyes commented Jul 24, 2019

Uh oh!

foreyes commented Jul 24, 2019

Uh oh!

Uh oh!

Uh oh!

foreyes commented Jul 24, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

foreyes commented Jul 24, 2019

Uh oh!

XuHuaiyu left a comment

Choose a reason for hiding this comment

Uh oh!

XuHuaiyu commented Jul 25, 2019

Uh oh!

Uh oh!

foreyes commented Aug 6, 2019

Uh oh!

XuHuaiyu Aug 6, 2019

Choose a reason for hiding this comment

Uh oh!

XuHuaiyu Aug 6, 2019

Choose a reason for hiding this comment

Uh oh!

foreyes Aug 6, 2019

Choose a reason for hiding this comment

Uh oh!

foreyes Aug 6, 2019

Choose a reason for hiding this comment

Uh oh!

foreyes commented Aug 6, 2019

Uh oh!

eurekaka left a comment

Choose a reason for hiding this comment

Uh oh!

XuHuaiyu left a comment

Choose a reason for hiding this comment

Uh oh!

foreyes commented Aug 7, 2019

Uh oh!

XuHuaiyu commented Aug 7, 2019

Uh oh!

foreyes commented Aug 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

foreyes commented Jul 22, 2019 •

edited

Loading

codecov Bot commented Jul 22, 2019 •

edited

Loading

foreyes commented Aug 7, 2019 •

edited

Loading