[opt](nereids)new way to set pre-agg status #34738

starocean999 · 2024-05-12T12:17:46Z

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-05-12T12:17:51Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

starocean999 · 2024-05-12T12:17:55Z

run buildall

starocean999 · 2024-05-14T08:05:51Z

run buildall

starocean999 · 2024-05-14T10:51:10Z

run buildall

doris-robot · 2024-05-14T11:33:49Z

TPC-H: Total hot run time: 41736 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 78c4fe282afaa03a8391b6dad456ac9ad74f948b, data reload: false

------ Round 1 ----------------------------------
q1	17599	4345	4204	4204
q2	2033	192	185	185
q3	10441	1290	1264	1264
q4	10199	798	821	798
q5	7533	2708	2688	2688
q6	228	130	136	130
q7	1036	605	585	585
q8	9227	2139	2103	2103
q9	10043	6829	6728	6728
q10	9279	3883	3854	3854
q11	442	273	237	237
q12	480	218	225	218
q13	17226	3159	3096	3096
q14	260	206	218	206
q15	520	470	478	470
q16	476	400	398	398
q17	973	737	717	717
q18	8438	7781	7950	7781
q19	3845	1584	1491	1491
q20	656	312	316	312
q21	5183	3995	4231	3995
q22	358	281	276	276
Total cold run time: 116475 ms
Total hot run time: 41736 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4491	4370	4357	4357
q2	371	288	281	281
q3	3114	2972	2753	2753
q4	1878	1550	1658	1550
q5	5502	5454	5488	5454
q6	217	126	126	126
q7	2292	1966	1980	1966
q8	3233	3390	3374	3374
q9	8579	8722	8577	8577
q10	3929	3766	3784	3766
q11	594	485	511	485
q12	769	612	620	612
q13	17113	3111	3107	3107
q14	308	291	285	285
q15	518	482	489	482
q16	483	457	446	446
q17	1743	1490	1466	1466
q18	7627	7574	7473	7473
q19	1620	1527	1543	1527
q20	1965	1764	1736	1736
q21	5025	4806	4740	4740
q22	566	502	464	464
Total cold run time: 71937 ms
Total hot run time: 55027 ms

doris-robot · 2024-05-14T11:45:07Z

TPC-DS: Total hot run time: 185998 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 78c4fe282afaa03a8391b6dad456ac9ad74f948b, data reload: false

query1	927	375	371	371
query2	6439	2424	2231	2231
query3	6642	202	202	202
query4	22929	21274	21206	21206
query5	4167	410	400	400
query6	256	172	172	172
query7	4590	293	292	292
query8	246	187	184	184
query9	8569	2384	2378	2378
query10	435	265	256	256
query11	14674	14322	14167	14167
query12	136	92	87	87
query13	1644	374	393	374
query14	9667	7937	6601	6601
query15	221	184	172	172
query16	7840	263	255	255
query17	1734	564	541	541
query18	2008	278	262	262
query19	198	140	146	140
query20	91	86	90	86
query21	194	124	125	124
query22	5113	4883	4832	4832
query23	33817	33480	33421	33421
query24	6697	2863	2883	2863
query25	518	364	363	363
query26	692	154	148	148
query27	1914	320	331	320
query28	3841	2077	2103	2077
query29	839	591	591	591
query30	251	179	185	179
query31	983	782	744	744
query32	86	50	52	50
query33	476	245	236	236
query34	872	470	477	470
query35	743	668	654	654
query36	1041	896	900	896
query37	103	71	68	68
query38	2890	2780	2827	2780
query39	1586	1551	1587	1551
query40	203	124	120	120
query41	46	44	42	42
query42	102	95	95	95
query43	564	554	550	550
query44	1033	719	733	719
query45	276	264	257	257
query46	1053	730	737	730
query47	1992	1951	1905	1905
query48	364	311	299	299
query49	776	415	425	415
query50	755	389	376	376
query51	6761	6721	6732	6721
query52	110	87	98	87
query53	346	285	288	285
query54	522	433	426	426
query55	78	73	73	73
query56	249	229	227	227
query57	1219	1153	1156	1153
query58	222	211	216	211
query59	3552	3233	3240	3233
query60	260	230	228	228
query61	90	87	92	87
query62	594	474	462	462
query63	304	276	284	276
query64	8240	7334	7356	7334
query65	3131	3065	3070	3065
query66	822	335	334	334
query67	15385	15174	15030	15030
query68	4522	539	529	529
query69	474	297	295	295
query70	1174	1107	1139	1107
query71	383	263	262	262
query72	7159	2560	2320	2320
query73	690	316	317	316
query74	6487	6200	6183	6183
query75	3240	2601	2612	2601
query76	2225	994	1020	994
query77	381	261	255	255
query78	10545	10404	10061	10061
query79	2562	530	568	530
query80	1051	429	436	429
query81	534	246	250	246
query82	972	98	105	98
query83	250	163	166	163
query84	245	90	89	89
query85	1022	267	266	266
query86	451	323	319	319
query87	3334	3102	3121	3102
query88	3736	2354	2336	2336
query89	479	363	385	363
query90	2031	189	186	186
query91	123	95	139	95
query92	62	47	48	47
query93	1835	519	505	505
query94	1130	179	187	179
query95	383	294	294	294
query96	585	268	267	267
query97	3201	2972	3003	2972
query98	242	213	213	213
query99	1168	918	892	892
Total cold run time: 267794 ms
Total hot run time: 185998 ms

doris-robot · 2024-05-14T11:50:38Z

ClickBench: Total hot run time: 30.06 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 78c4fe282afaa03a8391b6dad456ac9ad74f948b, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.66	0.07	0.07
query5	0.50	0.48	0.52
query6	1.11	0.71	0.71
query7	0.02	0.01	0.02
query8	0.04	0.04	0.05
query9	0.53	0.49	0.50
query10	0.54	0.56	0.54
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.59	0.59	0.60
query14	0.79	0.76	0.77
query15	0.82	0.80	0.80
query16	0.36	0.37	0.38
query17	0.96	0.99	0.97
query18	0.22	0.24	0.24
query19	1.87	1.67	1.72
query20	0.02	0.01	0.01
query21	15.51	0.69	0.68
query22	4.27	7.56	1.71
query23	18.29	1.28	1.23
query24	1.76	0.24	0.22
query25	0.16	0.08	0.08
query26	0.25	0.16	0.16
query27	0.08	0.08	0.07
query28	13.37	1.02	0.99
query29	13.27	3.28	3.25
query30	0.24	0.06	0.05
query31	2.85	0.38	0.38
query32	3.31	0.47	0.47
query33	2.83	2.86	2.80
query34	17.06	4.38	4.50
query35	4.43	4.46	4.67
query36	0.65	0.48	0.46
query37	0.18	0.14	0.15
query38	0.14	0.14	0.14
query39	0.05	0.04	0.03
query40	0.16	0.14	0.13
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.73 s
Total hot run time: 30.06 s

starocean999 · 2024-05-16T03:38:33Z

run buildall

starocean999 · 2024-05-16T10:11:07Z

run buildall

morrySnow · 2024-05-17T04:04:32Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java

+    PREAGG_STATUS_AGG_FILTER_SCAN(RuleTypeClass.REWRITE),
+    PREAGG_STATUS_AGG_PROJECT_SCAN(RuleTypeClass.REWRITE),
+    PREAGG_STATUS_AGG_PROJECT_FILTER_SCAN(RuleTypeClass.REWRITE),
+    PREAGG_STATUS_AGG_FILTER_PROJECT_SCAN(RuleTypeClass.REWRITE),


why we have filter above project? for variant?

morrySnow · 2024-05-17T04:18:21Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AdjustPreAggStatus.java

+        MaterializedIndexMeta meta = olapScan.getTable().getIndexMetaByIndexId(selectIndexId);
+        if (meta.getKeysType() == KeysType.DUP_KEYS || (meta.getKeysType() == KeysType.UNIQUE_KEYS
+                && olapScan.getTable().getEnableUniqueKeyMergeOnWrite())) {
+            return PreAggStatus.on();
+        }


let it be a short path method before we do extract and normalize agg function? for most case, this will work well

starocean999 · 2024-05-17T06:01:24Z

run buildall

github-actions · 2024-05-17T08:24:56Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-05-17T08:24:59Z

PR approved by anyone and no changes requested.

fix bug introduced by #34738 set preagg to off if agg function has no children( like count(*) )

fix bug introduced by apache#34738 set preagg to off if agg function has no children( like count(*) )

starocean999 marked this pull request as ready for review May 14, 2024 07:25

starocean999 marked this pull request as draft May 14, 2024 07:25

starocean999 force-pushed the nereids_1019 branch from 8f13fb0 to 4095719 Compare May 14, 2024 08:05

starocean999 force-pushed the nereids_1019 branch from 4095719 to 78c4fe2 Compare May 14, 2024 10:51

starocean999 force-pushed the nereids_1019 branch from 78c4fe2 to 7a9b62f Compare May 16, 2024 03:38

starocean999 marked this pull request as ready for review May 16, 2024 09:42

starocean999 force-pushed the nereids_1019 branch from 7a9b62f to 12988ae Compare May 16, 2024 10:10

morrySnow reviewed May 17, 2024

View reviewed changes

starocean999 added 5 commits May 17, 2024 12:39

[opt](nereids)new way to set pre-agg status

5662782

refactor

1dcd14a

fix fe ut

65060bd

remove unused code

550e07d

update code

2a4e324

starocean999 force-pushed the nereids_1019 branch from 12988ae to 2a4e324 Compare May 17, 2024 06:01

morrySnow approved these changes May 17, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 17, 2024

github-actions bot added the reviewed label May 17, 2024

morrySnow added the dev/2.1.x label May 17, 2024

morrySnow added the dev/3.0.x label May 17, 2024

englefly approved these changes May 20, 2024

View reviewed changes

starocean999 merged commit 27c75b3 into apache:master May 20, 2024

yiguolei pushed a commit that referenced this pull request May 21, 2024

[opt](nereids)new way to set pre-agg status (#34738)

f376232

morrySnow added dev/2.1.4-merged and removed dev/2.1.x labels May 23, 2024

starocean999 mentioned this pull request May 24, 2024

[fix](nereids)the preagg state for count(*) is wrong #35254

Merged

morrySnow pushed a commit that referenced this pull request May 24, 2024

[fix](nereids) the preagg state for count(*) is wrong (#35254)

31529db

fix bug introduced by #34738 set preagg to off if agg function has no children( like count(*) )

dataroaring pushed a commit that referenced this pull request May 26, 2024

[opt](nereids)new way to set pre-agg status (#34738)

139da0a

dataroaring pushed a commit that referenced this pull request May 26, 2024

[fix](nereids) the preagg state for count(*) is wrong (#35254)

833c5a7

fix bug introduced by #34738 set preagg to off if agg function has no children( like count(*) )

seawinde pushed a commit to seawinde/doris that referenced this pull request May 27, 2024

[fix](nereids) the preagg state for count(*) is wrong (apache#35254)

7245ae9

fix bug introduced by apache#34738 set preagg to off if agg function has no children( like count(*) )

gavinchou added dev/3.0.0-merged and removed dev/3.0.x labels Jun 11, 2024

ixzc mentioned this pull request Dec 5, 2024

[Bug] different select express leading to dozes of times performance difference #44353

Closed

3 tasks

[opt](nereids)new way to set pre-agg status #34738

[opt](nereids)new way to set pre-agg status #34738

Uh oh!

Conversation

starocean999 commented May 12, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented May 12, 2024

Uh oh!

starocean999 commented May 12, 2024

Uh oh!

starocean999 commented May 14, 2024

Uh oh!

starocean999 commented May 14, 2024

Uh oh!

doris-robot commented May 14, 2024

Uh oh!

doris-robot commented May 14, 2024

Uh oh!

doris-robot commented May 14, 2024

Uh oh!

starocean999 commented May 16, 2024

Uh oh!

starocean999 commented May 16, 2024

Uh oh!

morrySnow May 17, 2024

Choose a reason for hiding this comment

Uh oh!

morrySnow May 17, 2024

Choose a reason for hiding this comment

Uh oh!

starocean999 commented May 17, 2024

Uh oh!

github-actions bot commented May 17, 2024

Uh oh!

github-actions bot commented May 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants