expression, planner: fix decimal results for aggregate functions by dyzsr · Pull Request #20017 · pingcap/tidb

dyzsr · 2020-09-15T13:24:24Z

What problem does this PR solve?

Issue Number: close #19426, fix PR #19592

Problem Summary:

Execute

create table t(a int primary key, b int);
insert into t values (1, 11), (4, 44), (2, 22), (3, 33);

Obtained

mysql> select sum(case when a <= 0 or a > 1000 then 0.0 else b end) from t;
+-------------------------------------------------------+
| sum(case when a <= 0 or a > 1000 then 0.0 else b end) |
+-------------------------------------------------------+
|                                                   110 |
+-------------------------------------------------------+
1 row in set (0.01 sec)

mysql> select group_concat(case when a <= 0 or a > 1000 then 0.0 else b end) from t;
+----------------------------------------------------------------+
| group_concat(case when a <= 0 or a > 1000 then 0.0 else b end) |
+----------------------------------------------------------------+
| 11,22,33,44                                                    |
+----------------------------------------------------------------+
1 row in set (0.00 sec)

mysql> select distinct case when a <= 0 or a > 1000 then 0.0 else b end from t;
+--------------------------------------------------+
| case when a <= 0 or a > 1000 then 0.0 else b end |
+--------------------------------------------------+
|                                               11 |
|                                               22 |
|                                               33 |
|                                               44 |
+--------------------------------------------------+
4 rows in set (0.00 sec)

Expect

mysql> select sum(case when a <= 0 or a > 1000 then 0.0 else b end) from t;
+-------------------------------------------------------+
| sum(case when a <= 0 or a > 1000 then 0.0 else b end) |
+-------------------------------------------------------+
|                                                 110.0 |
+-------------------------------------------------------+
1 row in set (0.00 sec)

mysql>  select group_concat(case when a <= 0 or a > 1000 then 0.0 else b end) from t;
+----------------------------------------------------------------+
| group_concat(case when a <= 0 or a > 1000 then 0.0 else b end) |
+----------------------------------------------------------------+
| 11.0,22.0,33.0,44.0                                            |
+----------------------------------------------------------------+
1 row in set (0.00 sec)

mysql> select distinct case when a <= 0 or a > 1000 then 0.0 else b end from t;
+--------------------------------------------------+
| case when a <= 0 or a > 1000 then 0.0 else b end |
+--------------------------------------------------+
|                                             11.0 |
|                                             22.0 |
|                                             33.0 |
|                                             44.0 |
+--------------------------------------------------+
4 rows in set (0.00 sec)

What is changed and how it works?

wrap cast as decimal for arguments of group_concat to preserve fraction part digits of the results.
round the decimal final result in AppendFinalResult2Chunk to the argument's precision.

Related changes

Need to cherry-pick to the release branch 3.0, 4.0

Check List

Tests

Unit test
Integration test

Side effects

Performance regression
- Consumes more CPU
- Consumes more MEM

Release note

fix the fraction part of decimal results for aggregate functions

dyzsr · 2020-09-15T15:26:26Z

/run-all-tests

dyzsr · 2020-09-16T03:41:55Z

/run-all-tests

dyzsr · 2020-09-16T08:29:00Z

PTAL @wjhuang2016

SunRunAway · 2020-09-18T04:25:20Z


+	"github.com/cznic/mathutil"
+	"github.com/pingcap/parser/mysql"
+


Please remove the empty line

SunRunAway · 2020-09-18T04:46:26Z

+func (a *baseFuncDesc) WrapCastAsDecimalForAggArgs(ctx sessionctx.Context) {
+	if _, ok := needCastAsDecimalAggFuncs[a.Name]; ok {
+		for i := range a.Args {
+			if tp := a.Args[i].GetType(); tp.Tp == mysql.TypeNewDecimal {


The last argument of group_cancat its separator, is it ok?

SunRunAway · 2020-09-18T06:24:57Z

+func (a *baseFuncDesc) WrapCastAsDecimalForAggArgs(ctx sessionctx.Context) {
+	if a.Name == ast.AggFuncGroupConcat {
+		if tp := a.Args[0].GetType(); tp.Tp == mysql.TypeNewDecimal {
+			a.Args[0] = expression.BuildCastFunction(ctx, a.Args[0], tp)


What if the second argument needs to be cast?

select group_concat(a, case when a <= 2 or a > 1000 then 0.0 else b end) from t;

qw4990 · 2020-11-04T09:00:27Z

/run-all-tests

qw4990

LGTM

dyzsr · 2020-11-11T03:08:20Z

/run-all-tests

lzmhhh123 · 2020-11-11T05:51:59Z

/run-common-test tidb-test=pr/1100
/run-integration-common-test tidb-test=pr/1100

lzmhhh123 · 2020-11-11T06:11:22Z

/merge

ti-srebot · 2020-11-11T06:13:34Z

/run-all-tests

ti-srebot · 2020-11-11T06:16:32Z

@dyzsr merge failed.

lzmhhh123 · 2020-11-11T06:17:58Z

/run-all-tests

lzmhhh123 · 2020-11-11T06:19:20Z

/merge

ti-srebot · 2020-11-11T06:19:31Z

/run-all-tests

ti-srebot · 2020-11-11T06:40:28Z

@dyzsr merge failed.

lzmhhh123 · 2020-11-11T06:49:57Z

/run-cherry-picker

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2020-11-11T06:51:08Z

cherry pick to release-4.0 in PR #20984

) (#20984) * cherry pick #20017 to release-4.0 Signed-off-by: ti-srebot <ti-srebot@pingcap.com> * resolve conflicts Co-authored-by: dy <34701401+dyzsr@users.noreply.github.com> Co-authored-by: dongyan <dybest@icloud.com> Co-authored-by: Yuanjia Zhang <zhangyuanjia@pingcap.com> Co-authored-by: Yiding Cui <winoros@gmail.com>

expression, planner: wrap cast as decimal for aggregate function args

74c55dc

dyzsr requested review from a team as code owners September 15, 2020 13:24

dyzsr requested review from SunRunAway and qw4990 and removed request for a team September 15, 2020 13:24

github-actions Bot added the sig/execution SIG execution label Sep 15, 2020

dyzsr marked this pull request as draft September 15, 2020 13:37

dyzsr and others added 3 commits September 15, 2020 22:00

only wrap for scalar function argument case, if, ifnull

cdfeebe

fix

9b7ebb8

Merge branch 'master' into aggdecimalfrac

fe0f880

dyzsr marked this pull request as ready for review September 15, 2020 15:24

dyzsr and others added 2 commits September 16, 2020 11:17

Merge branch 'master' into aggdecimalfrac

f5497dd

add Round in AppendFinalResult2Chunk

c8ee654

dyzsr changed the title ~~expression, planner: wrap cast as decimal for aggregate function arguments~~ expression, planner: fix decimal results for aggregate functions Sep 16, 2020

Merge branch 'master' into aggdecimalfrac

54a1b7b

winoros reviewed Sep 17, 2020

View reviewed changes

Comment thread expression/aggregation/base_func.go Outdated

ti-srebot added challenge-program difficulty/easy labels Sep 18, 2020

SunRunAway reviewed Sep 18, 2020

View reviewed changes

dyzsr and others added 3 commits September 18, 2020 13:27

remove empty lines

78f230a

remove needCastAsDecimalAggFuncs

d318ef1

Merge branch 'master' into aggdecimalfrac

2419d28

SunRunAway reviewed Sep 18, 2020

View reviewed changes

fix

8cd5ab1

qw4990 approved these changes Nov 4, 2020

View reviewed changes

ti-srebot removed the status/LGT1 Indicates that a PR has LGTM 1. label Nov 4, 2020

ti-srebot approved these changes Nov 4, 2020

View reviewed changes

ti-srebot added the status/LGT2 Indicates that a PR has LGTM 2. label Nov 4, 2020

Merge branch 'master' into aggdecimalfrac

136d19b

ti-srebot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 11, 2020

Merge branch 'master' into aggdecimalfrac

16d22ba

lzmhhh123 merged commit a3facd0 into pingcap:master Nov 11, 2020

lzmhhh123 added the needs-cherry-pick-4.0 label Nov 11, 2020

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Nov 11, 2020

cherry pick pingcap#20017 to release-4.0

acdb935

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot mentioned this pull request Nov 11, 2020

expression, planner: fix decimal results for aggregate functions (#20017) #20984

Merged

fzhedu mentioned this pull request Nov 26, 2021

executor: avoid sum from avg overflow #30010

Merged

12 tasks


		"github.com/cznic/mathutil"
		"github.com/pingcap/parser/mysql"

Conversation

dyzsr commented Sep 15, 2020 • edited by ti-srebot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

Uh oh!

dyzsr commented Sep 15, 2020

Uh oh!

dyzsr commented Sep 16, 2020

Uh oh!

dyzsr commented Sep 16, 2020

Uh oh!

Uh oh!

SunRunAway Sep 18, 2020

Choose a reason for hiding this comment

Uh oh!

SunRunAway Sep 18, 2020

Choose a reason for hiding this comment

Uh oh!

SunRunAway Sep 18, 2020

Choose a reason for hiding this comment

Uh oh!

qw4990 commented Nov 4, 2020

Uh oh!

qw4990 left a comment

Choose a reason for hiding this comment

Uh oh!

dyzsr commented Nov 11, 2020

Uh oh!

lzmhhh123 commented Nov 11, 2020

Uh oh!

lzmhhh123 commented Nov 11, 2020

Uh oh!

ti-srebot commented Nov 11, 2020

Uh oh!

ti-srebot commented Nov 11, 2020

Uh oh!

lzmhhh123 commented Nov 11, 2020

Uh oh!

lzmhhh123 commented Nov 11, 2020

Uh oh!

ti-srebot commented Nov 11, 2020

Uh oh!

ti-srebot commented Nov 11, 2020

Uh oh!

lzmhhh123 commented Nov 11, 2020

Uh oh!

ti-srebot commented Nov 11, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

dyzsr commented Sep 15, 2020 •

edited by ti-srebot

Loading