Skip to content

Conversation

@keanji-x
Copy link
Contributor

Proposed changes

This PR introduces a change in the method removeOneSumLiteral to enhance the performance of sum literal rewriting in SQL queries. The modification ensures that sum literals appearing only once, such as in expressions like select count(id1 + 1), count(id2 + 1) from t, are not rewritten.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@keanji-x
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41036 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2b10d1a4be19ab4883fbd939ea708d1cd723ef5f, data reload: false

------ Round 1 ----------------------------------
q1	17598	4229	4189	4189
q2	2018	200	199	199
q3	10434	1246	1250	1246
q4	10185	858	775	775
q5	7468	2749	2665	2665
q6	217	132	141	132
q7	956	602	646	602
q8	9212	2133	2051	2051
q9	9410	6698	6649	6649
q10	9336	3918	3895	3895
q11	451	250	248	248
q12	425	233	237	233
q13	17546	3200	3147	3147
q14	262	223	213	213
q15	501	476	491	476
q16	538	402	400	400
q17	962	691	682	682
q18	8362	7755	7790	7755
q19	4980	1562	1531	1531
q20	634	326	318	318
q21	5134	3357	4081	3357
q22	337	275	273	273
Total cold run time: 116966 ms
Total hot run time: 41036 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4508	4397	4393	4393
q2	378	287	279	279
q3	3207	2900	2899	2899
q4	1870	1605	1622	1605
q5	5491	5487	5504	5487
q6	216	128	126	126
q7	2176	1808	1791	1791
q8	3231	3379	3334	3334
q9	8504	8612	8603	8603
q10	4081	3654	3738	3654
q11	608	507	507	507
q12	805	618	624	618
q13	16345	3129	3170	3129
q14	307	272	260	260
q15	529	510	492	492
q16	482	418	451	418
q17	1787	1497	1517	1497
q18	7794	7615	7460	7460
q19	3758	1569	1565	1565
q20	1985	1758	1768	1758
q21	10718	4691	4595	4595
q22	571	488	479	479
Total cold run time: 79351 ms
Total hot run time: 54949 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169301 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2b10d1a4be19ab4883fbd939ea708d1cd723ef5f, data reload: false

query1	934	394	375	375
query2	6437	2474	2426	2426
query3	6639	205	203	203
query4	19237	17413	17314	17314
query5	4192	435	424	424
query6	252	164	163	163
query7	4581	299	286	286
query8	244	181	182	181
query9	8616	2368	2346	2346
query10	455	273	259	259
query11	10545	10173	9994	9994
query12	135	89	88	88
query13	1642	374	368	368
query14	10114	6903	7404	6903
query15	207	164	167	164
query16	7160	273	272	272
query17	1013	553	541	541
query18	1920	289	275	275
query19	217	197	151	151
query20	88	85	89	85
query21	199	131	137	131
query22	4228	3912	3887	3887
query23	33676	33179	33117	33117
query24	5124	2865	2866	2865
query25	461	354	365	354
query26	692	153	161	153
query27	1833	315	328	315
query28	3778	2037	2023	2023
query29	834	599	597	597
query30	228	148	147	147
query31	919	781	737	737
query32	60	51	54	51
query33	421	271	262	262
query34	873	476	473	473
query35	711	613	592	592
query36	1046	946	933	933
query37	103	70	66	66
query38	2889	2765	2785	2765
query39	844	794	827	794
query40	193	123	124	123
query41	45	43	45	43
query42	101	94	94	94
query43	575	548	536	536
query44	1104	716	742	716
query45	179	163	167	163
query46	1043	724	700	700
query47	1825	1779	1774	1774
query48	362	301	296	296
query49	766	376	387	376
query50	770	394	389	389
query51	6926	6908	6830	6830
query52	106	88	89	88
query53	354	275	280	275
query54	551	428	421	421
query55	73	74	72	72
query56	262	241	236	236
query57	1103	1065	1040	1040
query58	230	219	216	216
query59	3432	3289	3131	3131
query60	278	264	286	264
query61	91	87	88	87
query62	553	449	444	444
query63	312	284	281	281
query64	2535	1722	1785	1722
query65	3165	3101	3091	3091
query66	661	327	321	321
query67	14972	14907	15037	14907
query68	4572	537	523	523
query69	446	271	265	265
query70	1133	1099	1076	1076
query71	453	270	265	265
query72	7788	2697	2479	2479
query73	710	319	318	318
query74	6007	5680	5546	5546
query75	3377	2647	2581	2581
query76	2345	1013	996	996
query77	392	260	266	260
query78	10274	9747	9744	9744
query79	1648	511	508	508
query80	702	444	437	437
query81	453	224	216	216
query82	784	93	90	90
query83	187	165	167	165
query84	265	84	83	83
query85	910	275	257	257
query86	443	295	295	295
query87	3303	3141	3172	3141
query88	3883	2337	2326	2326
query89	461	394	385	385
query90	2075	188	190	188
query91	123	94	97	94
query92	60	48	51	48
query93	2439	516	496	496
query94	1206	205	187	187
query95	394	314	318	314
query96	602	269	268	268
query97	3161	2976	3062	2976
query98	232	220	210	210
query99	1229	848	878	848
Total cold run time: 248171 ms
Total hot run time: 169301 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.44 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2b10d1a4be19ab4883fbd939ea708d1cd723ef5f, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.06	0.08
query5	0.49	0.48	0.50
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.55	0.49	0.50
query10	0.54	0.56	0.54
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.59	0.59	0.60
query14	0.79	0.80	0.77
query15	0.83	0.82	0.82
query16	0.37	0.37	0.37
query17	0.95	0.98	1.01
query18	0.25	0.22	0.26
query19	1.83	1.81	1.78
query20	0.01	0.01	0.01
query21	15.50	0.70	0.68
query22	4.57	6.82	1.69
query23	18.25	1.49	1.26
query24	1.73	0.26	0.22
query25	0.13	0.08	0.09
query26	0.26	0.17	0.16
query27	0.07	0.08	0.08
query28	13.39	1.03	1.00
query29	13.21	3.37	3.35
query30	0.24	0.06	0.05
query31	2.85	0.38	0.39
query32	3.27	0.48	0.47
query33	2.93	2.86	2.89
query34	17.01	4.43	4.41
query35	4.50	4.47	4.57
query36	0.67	0.50	0.46
query37	0.18	0.15	0.15
query38	0.15	0.14	0.14
query39	0.05	0.03	0.03
query40	0.17	0.14	0.14
query41	0.09	0.04	0.04
query42	0.06	0.05	0.05
query43	0.04	0.03	0.04
Total cold run time: 110.03 s
Total hot run time: 30.44 s

@keanji-x
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39460 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dfdf88b8f2520507e063f894f1a2fca015b90430, data reload: false

------ Round 1 ----------------------------------
q1	17631	4301	4194	4194
q2	2039	190	191	190
q3	10452	1128	1090	1090
q4	10199	838	713	713
q5	7474	2640	2708	2640
q6	222	134	134	134
q7	940	605	607	605
q8	9224	2050	2075	2050
q9	8924	6498	6437	6437
q10	8979	3692	3735	3692
q11	436	233	242	233
q12	515	231	226	226
q13	18792	2963	2943	2943
q14	257	219	209	209
q15	528	474	471	471
q16	502	386	366	366
q17	957	636	669	636
q18	7956	7494	7406	7406
q19	6546	1435	1481	1435
q20	646	339	322	322
q21	4800	3186	3144	3144
q22	369	332	324	324
Total cold run time: 118388 ms
Total hot run time: 39460 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4331	4250	4199	4199
q2	380	265	263	263
q3	2954	2911	2850	2850
q4	2001	1719	1754	1719
q5	5544	5513	5431	5431
q6	219	130	133	130
q7	2218	1868	1859	1859
q8	3297	3431	3408	3408
q9	8683	8651	8846	8651
q10	4062	3895	3684	3684
q11	595	488	505	488
q12	799	629	651	629
q13	15914	3194	3122	3122
q14	310	261	272	261
q15	554	475	475	475
q16	499	433	410	410
q17	1798	1532	1521	1521
q18	8058	8041	7839	7839
q19	1817	1644	1640	1640
q20	3013	1883	1849	1849
q21	8923	4817	4800	4800
q22	604	537	556	537
Total cold run time: 76573 ms
Total hot run time: 55765 ms

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 21, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 40259 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dfdf88b8f2520507e063f894f1a2fca015b90430, data reload: false

------ Round 1 ----------------------------------
q1	17633	4409	4247	4247
q2	2028	187	202	187
q3	10448	1156	1093	1093
q4	10188	837	741	741
q5	7483	2628	2660	2628
q6	225	138	140	138
q7	958	607	600	600
q8	9218	2073	2044	2044
q9	9053	6504	6421	6421
q10	8987	3719	3751	3719
q11	466	240	246	240
q12	455	224	218	218
q13	18051	2957	2976	2957
q14	265	219	213	213
q15	522	482	470	470
q16	539	380	379	379
q17	957	729	713	713
q18	8049	7378	7328	7328
q19	9218	1434	1531	1434
q20	701	306	329	306
q21	4956	3846	3913	3846
q22	385	337	343	337
Total cold run time: 120785 ms
Total hot run time: 40259 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4398	4242	4227	4227
q2	384	284	288	284
q3	3174	2880	2887	2880
q4	1978	1700	1764	1700
q5	5484	5427	5513	5427
q6	232	133	145	133
q7	2271	1830	1854	1830
q8	3310	3408	3409	3408
q9	8671	8814	8714	8714
q10	4069	3706	3841	3706
q11	583	497	493	493
q12	815	607	621	607
q13	16391	3136	3142	3136
q14	296	287	281	281
q15	509	490	489	489
q16	522	442	438	438
q17	1846	1531	1497	1497
q18	8038	7734	7949	7734
q19	1854	1775	1684	1684
q20	3057	1884	1844	1844
q21	5038	4884	4732	4732
q22	615	530	546	530
Total cold run time: 73535 ms
Total hot run time: 55774 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173638 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dfdf88b8f2520507e063f894f1a2fca015b90430, data reload: false

query1	933	386	375	375
query2	6446	2486	2417	2417
query3	6637	207	209	207
query4	19240	17323	17210	17210
query5	3650	495	474	474
query6	239	165	159	159
query7	4597	307	292	292
query8	329	299	293	293
query9	8452	2445	2445	2445
query10	605	295	298	295
query11	10348	10047	9960	9960
query12	126	95	87	87
query13	1647	378	367	367
query14	10249	7465	7839	7465
query15	265	208	199	199
query16	7833	276	270	270
query17	1862	553	535	535
query18	2008	281	298	281
query19	210	156	164	156
query20	104	86	87	86
query21	200	147	129	129
query22	4523	4296	4430	4296
query23	33935	33745	33393	33393
query24	11003	2917	2779	2779
query25	602	361	362	361
query26	941	151	155	151
query27	2243	310	311	310
query28	6885	2064	2049	2049
query29	897	617	601	601
query30	256	152	150	150
query31	936	743	740	740
query32	92	53	56	53
query33	754	281	275	275
query34	896	474	455	455
query35	742	626	627	626
query36	1104	948	956	948
query37	140	72	74	72
query38	2916	2727	2745	2727
query39	850	819	832	819
query40	217	130	130	130
query41	56	55	55	55
query42	129	98	107	98
query43	606	561	561	561
query44	1246	745	749	745
query45	197	170	171	170
query46	1090	716	744	716
query47	1874	1757	1794	1757
query48	392	303	302	302
query49	861	421	461	421
query50	780	389	394	389
query51	6995	6793	6770	6770
query52	110	95	95	95
query53	368	292	289	289
query54	909	470	453	453
query55	75	76	73	73
query56	302	287	330	287
query57	1136	1058	1021	1021
query58	255	258	245	245
query59	3408	3086	3258	3086
query60	306	279	275	275
query61	97	94	89	89
query62	600	446	450	446
query63	322	303	295	295
query64	8804	2327	1756	1756
query65	3158	3099	3105	3099
query66	747	330	328	328
query67	15518	15046	14899	14899
query68	4560	547	528	528
query69	574	497	388	388
query70	1195	1132	1102	1102
query71	400	283	284	283
query72	7197	5049	5398	5049
query73	783	318	316	316
query74	5906	5575	5461	5461
query75	3442	2677	2644	2644
query76	2267	957	945	945
query77	654	310	296	296
query78	10404	9861	9671	9671
query79	2264	508	514	508
query80	1580	461	468	461
query81	580	221	223	221
query82	762	105	102	102
query83	203	170	168	168
query84	277	83	85	83
query85	1203	282	280	280
query86	408	345	331	331
query87	3318	3081	3077	3077
query88	3891	2344	2331	2331
query89	489	376	370	370
query90	1803	194	188	188
query91	129	99	100	99
query92	61	50	49	49
query93	1934	497	492	492
query94	1040	184	186	184
query95	401	312	309	309
query96	599	272	264	264
query97	3285	3060	3045	3045
query98	220	200	198	198
query99	1229	851	838	838
Total cold run time: 269311 ms
Total hot run time: 173638 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.33 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dfdf88b8f2520507e063f894f1a2fca015b90430, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.24	0.06	0.06
query4	1.65	0.07	0.08
query5	0.51	0.49	0.49
query6	1.12	0.71	0.72
query7	0.02	0.01	0.01
query8	0.06	0.04	0.04
query9	0.54	0.48	0.50
query10	0.55	0.55	0.54
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.61
query14	0.77	0.76	0.79
query15	0.84	0.81	0.80
query16	0.36	0.35	0.38
query17	0.97	1.05	1.02
query18	0.22	0.26	0.24
query19	1.77	1.73	1.69
query20	0.01	0.01	0.01
query21	15.42	0.67	0.65
query22	4.78	7.50	1.75
query23	18.28	1.44	1.25
query24	2.06	0.24	0.22
query25	0.16	0.09	0.08
query26	0.27	0.18	0.17
query27	0.08	0.07	0.08
query28	13.25	1.01	0.98
query29	12.62	3.29	3.23
query30	0.25	0.07	0.06
query31	2.86	0.38	0.39
query32	3.27	0.47	0.46
query33	2.92	2.91	2.89
query34	17.00	4.42	4.43
query35	4.51	4.50	4.56
query36	0.65	0.46	0.46
query37	0.20	0.16	0.17
query38	0.16	0.14	0.14
query39	0.05	0.03	0.03
query40	0.17	0.13	0.14
query41	0.09	0.05	0.05
query42	0.06	0.04	0.04
query43	0.05	0.05	0.04
Total cold run time: 109.8 s
Total hot run time: 30.33 s

@keanji-x keanji-x merged commit 9b5a764 into apache:master Jun 21, 2024
iszhangpch pushed a commit to iszhangpch/doris-p that referenced this pull request Jun 21, 2024
…stances (apache#35559)

## Proposed changes

This PR introduces a change in the method removeOneSumLiteral to enhance
the performance of sum literal rewriting in SQL queries. The
modification ensures that sum literals appearing only once, such as in
expressions like select count(id1 + 1), count(id2 + 1) from t, are not
rewritten.
dataroaring pushed a commit that referenced this pull request Jun 21, 2024
…stances (#35559)

## Proposed changes

This PR introduces a change in the method removeOneSumLiteral to enhance
the performance of sum literal rewriting in SQL queries. The
modification ensures that sum literals appearing only once, such as in
expressions like select count(id1 + 1), count(id2 + 1) from t, are not
rewritten.
keanji-x added a commit to keanji-x/doris that referenced this pull request Jul 1, 2024
…stances (apache#35559)

## Proposed changes

This PR introduces a change in the method removeOneSumLiteral to enhance
the performance of sum literal rewriting in SQL queries. The
modification ensures that sum literals appearing only once, such as in
expressions like select count(id1 + 1), count(id2 + 1) from t, are not
rewritten.
morrySnow pushed a commit that referenced this pull request Jul 1, 2024
…stances (#35559) (#37047)

pick from master #35559

This PR introduces a change in the method removeOneSumLiteral to enhance
the performance of sum literal rewriting in SQL queries. The
modification ensures that sum literals appearing only once, such as in
expressions like select count(id1 + 1), count(id2 + 1) from t, are not
rewritten.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.0-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants