Skip to content

Conversation

@xiaokang
Copy link
Contributor

@xiaokang xiaokang commented Feb 18, 2024

Proposed changes

pick from master #31075

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@xiaokang
Copy link
Contributor Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 50810 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dc7b60f64ec241516f2ab618a8ebd7175e1626f2, data reload: false

------ Round 1 ----------------------------------
q1	17683	4548	4428	4428
q2	2052	151	137	137
q3	10540	1871	1949	1871
q4	10408	1266	1324	1266
q5	9010	4025	3980	3980
q6	227	121	121	121
q7	2042	1627	1612	1612
q8	9466	2749	2741	2741
q9	13222	10580	10547	10547
q10	8857	3731	3698	3698
q11	424	249	258	249
q12	498	311	321	311
q13	19086	4214	4391	4214
q14	362	361	330	330
q15	517	459	470	459
q16	752	618	634	618
q17	1169	972	937	937
q18	7522	6953	6990	6953
q19	1690	1587	1534	1534
q20	525	324	311	311
q21	4461	4123	4098	4098
q22	493	395	401	395
Total cold run time: 121006 ms
Total hot run time: 50810 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4338	4271	4297	4271
q2	324	245	230	230
q3	4191	4157	4153	4153
q4	2735	2732	2742	2732
q5	7322	7179	7292	7179
q6	240	116	117	116
q7	3241	2813	2875	2813
q8	4358	4468	4527	4468
q9	17111	16976	16826	16826
q10	4242	4265	4298	4265
q11	756	673	664	664
q12	1021	845	877	845
q13	6727	3761	3724	3724
q14	450	427	438	427
q15	499	452	460	452
q16	742	697	691	691
q17	3832	3911	3854	3854
q18	8870	8938	8727	8727
q19	1712	1700	1663	1663
q20	2386	2155	2135	2135
q21	8517	8498	8516	8498
q22	1032	960	917	917
Total cold run time: 84646 ms
Total hot run time: 79650 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.83% (8035/21240)
Line Coverage: 29.50% (65516/222124)
Region Coverage: 28.97% (33716/116384)
Branch Coverage: 24.83% (17309/69724)
Coverage Report: http://coverage.selectdb-in.cc/coverage/dc7b60f64ec241516f2ab618a8ebd7175e1626f2_dc7b60f64ec241516f2ab618a8ebd7175e1626f2/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 243696 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dc7b60f64ec241516f2ab618a8ebd7175e1626f2, data reload: false

query1	930	386	388	386
query2	6516	2233	2157	2157
query3	6908	205	204	204
query4	20105	18001	18062	18001
query5	19723	6576	6526	6526
query6	303	217	238	217
query7	4146	298	302	298
query8	283	261	228	228
query9	3162	2727	2663	2663
query10	420	280	299	280
query11	11335	10833	10667	10667
query12	113	82	72	72
query13	5576	625	623	623
query14	18037	13373	13329	13329
query15	360	231	234	231
query16	6424	256	261	256
query17	1741	1455	863	863
query18	2322	410	415	410
query19	198	144	142	142
query20	78	78	74	74
query21	187	93	95	93
query22	5395	4897	5067	4897
query23	32547	31846	31920	31846
query24	6949	6480	6517	6480
query25	521	437	416	416
query26	528	163	157	157
query27	1864	291	290	290
query28	6115	2223	2191	2191
query29	2933	2654	2672	2654
query30	239	161	161	161
query31	916	742	724	724
query32	65	63	57	57
query33	394	240	254	240
query34	853	454	474	454
query35	1084	887	909	887
query36	1831	1583	1787	1583
query37	86	65	59	59
query38	3100	2968	2942	2942
query39	1366	1309	1340	1309
query40	187	93	95	93
query41	33	31	35	31
query42	88	83	86	83
query43	581	607	639	607
query44	1118	700	706	700
query45	237	230	227	227
query46	1232	971	957	957
query47	1793	1836	1799	1799
query48	986	671	661	661
query49	608	370	360	360
query50	866	575	610	575
query51	5576	5433	5480	5433
query52	100	73	82	73
query53	436	319	313	313
query54	2640	2464	2480	2464
query55	77	80	77	77
query56	189	200	196	196
query57	1236	1142	1190	1142
query58	219	201	201	201
query59	3433	3479	3092	3092
query60	203	181	183	181
query61	82	82	82	82
query62	769	471	497	471
query63	468	325	319	319
query64	2456	1482	1389	1389
query65	3630	3898	3580	3580
query66	774	357	355	355
query67	16185	15664	16258	15664
query68	6066	627	637	627
query69	542	350	365	350
query70	1837	1700	1988	1700
query71	402	301	311	301
query72	6423	3409	3424	3409
query73	726	316	318	316
query74	6307	5819	5866	5819
query75	4581	3809	3820	3809
query76	3754	1188	1170	1170
query77	552	244	246	244
query78	30836	52012	52458	52012
query79	15572	674	659	659
query80	4995	388	401	388
query81	558	231	236	231
query82	1135	94	100	94
query83	331	138	135	135
query84	259	68	69	68
query85	1992	280	276	276
query86	484	371	367	367
query87	3247	3024	3025	3024
query88	6675	2331	2321	2321
query89	442	316	305	305
query90	2492	205	219	205
query91	160	113	119	113
query92	62	53	50	50
query93	5267	543	530	530
query94	1846	205	210	205
query95	1105	1054	1048	1048
query96	648	334	321	321
query97	6451	6369	6393	6369
query98	189	184	169	169
query99	3845	869	920	869
Total cold run time: 341636 ms
Total hot run time: 243696 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.02 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dc7b60f64ec241516f2ab618a8ebd7175e1626f2, data reload: false

query1	0.02	0.02	0.03
query2	0.06	0.02	0.02
query3	0.25	0.04	0.05
query4	1.82	0.07	0.07
query5	0.54	0.53	0.51
query6	1.29	0.61	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.55	0.49	0.49
query10	0.55	0.52	0.53
query11	0.12	0.08	0.09
query12	0.11	0.09	0.10
query13	0.62	0.62	0.61
query14	0.77	0.79	0.79
query15	0.77	0.76	0.76
query16	0.37	0.36	0.37
query17	1.01	1.01	0.97
query18	0.23	0.25	0.26
query19	1.86	1.86	1.87
query20	0.01	0.01	0.01
query21	15.48	0.55	0.57
query22	2.48	2.08	1.37
query23	17.05	1.06	0.99
query24	6.10	1.70	1.36
query25	1.62	0.12	0.10
query26	0.35	0.14	0.14
query27	0.11	0.11	0.11
query28	5.92	0.79	0.74
query29	12.78	2.26	2.28
query30	0.56	0.52	0.54
query31	2.82	0.36	0.38
query32	3.42	0.50	0.50
query33	3.06	3.07	3.04
query34	15.26	4.80	4.80
query35	4.85	4.85	4.85
query36	1.06	1.02	1.00
query37	0.06	0.05	0.04
query38	0.03	0.02	0.03
query39	0.02	0.01	0.01
query40	0.16	0.14	0.14
query41	0.06	0.01	0.01
query42	0.02	0.02	0.01
query43	0.02	0.01	0.01
Total cold run time: 104.28 s
Total hot run time: 31.02 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit dc7b60f64ec241516f2ab618a8ebd7175e1626f2 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

@xiaokang
Copy link
Contributor Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 49732 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 32ce302111af3695d834a1142fa03881d03bee02, data reload: false

------ Round 1 ----------------------------------
q1	18103	4445	4367	4367
q2	2019	149	149	149
q3	10429	1860	1931	1860
q4	10305	1244	1316	1244
q5	8830	3904	4023	3904
q6	229	123	120	120
q7	2052	1594	1613	1594
q8	9265	2709	2728	2709
q9	10829	10458	10345	10345
q10	8654	3577	3525	3525
q11	412	232	236	232
q12	458	298	298	298
q13	18348	3945	3992	3945
q14	344	330	320	320
q15	513	454	471	454
q16	692	587	590	587
q17	1122	1016	989	989
q18	7380	6794	7026	6794
q19	1678	1574	1505	1505
q20	505	300	309	300
q21	4444	4128	4096	4096
q22	507	418	395	395
Total cold run time: 117118 ms
Total hot run time: 49732 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4384	4342	4323	4323
q2	316	223	223	223
q3	4207	4119	4166	4119
q4	2764	2824	2755	2755
q5	7288	7275	7204	7204
q6	235	121	117	117
q7	3248	2796	2845	2796
q8	4376	4506	4497	4497
q9	17200	17010	17068	17010
q10	4190	4216	4278	4216
q11	750	678	678	678
q12	1032	883	861	861
q13	7454	3750	3771	3750
q14	436	420	422	420
q15	490	462	466	462
q16	765	704	704	704
q17	3879	3848	3820	3820
q18	8891	8922	8740	8740
q19	1689	1714	1630	1630
q20	2421	2127	2165	2127
q21	8545	8524	8510	8510
q22	1023	939	929	929
Total cold run time: 85583 ms
Total hot run time: 79891 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.83% (8036/21240)
Line Coverage: 29.50% (65516/222122)
Region Coverage: 28.97% (33715/116380)
Branch Coverage: 24.82% (17305/69720)
Coverage Report: http://coverage.selectdb-in.cc/coverage/32ce302111af3695d834a1142fa03881d03bee02_32ce302111af3695d834a1142fa03881d03bee02/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 239459 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 32ce302111af3695d834a1142fa03881d03bee02, data reload: false

query1	935	383	381	381
query2	6519	2290	2079	2079
query3	6917	204	205	204
query4	19837	17945	17959	17945
query5	19702	6477	6474	6474
query6	283	215	223	215
query7	4157	302	289	289
query8	244	261	220	220
query9	3087	2725	2663	2663
query10	403	290	296	290
query11	11269	10739	10762	10739
query12	117	78	72	72
query13	5565	637	612	612
query14	17646	13825	13298	13298
query15	388	228	233	228
query16	6430	260	260	260
query17	1776	1468	858	858
query18	2337	401	400	400
query19	202	139	148	139
query20	73	79	73	73
query21	192	96	96	96
query22	5413	5168	5226	5168
query23	32510	32117	32042	32042
query24	6944	6536	6432	6432
query25	531	405	401	401
query26	525	163	157	157
query27	1873	295	294	294
query28	6109	2230	2192	2192
query29	2909	2776	2622	2622
query30	240	158	160	158
query31	922	706	717	706
query32	61	60	62	60
query33	400	263	247	247
query34	851	472	475	472
query35	1127	889	911	889
query36	1446	1473	1516	1473
query37	89	57	58	57
query38	3051	2928	2944	2928
query39	1354	1296	1317	1296
query40	201	94	94	94
query41	36	33	30	30
query42	94	75	83	75
query43	591	557	540	540
query44	1094	715	713	713
query45	239	226	224	224
query46	1230	973	965	965
query47	1824	1887	1712	1712
query48	945	689	659	659
query49	627	364	359	359
query50	876	614	595	595
query51	5503	5375	5380	5375
query52	94	77	76	76
query53	433	309	312	309
query54	2668	2478	2455	2455
query55	91	78	80	78
query56	222	206	202	202
query57	1237	1073	1057	1057
query58	197	203	199	199
query59	3591	3273	3283	3273
query60	196	186	208	186
query61	82	86	79	79
query62	799	486	477	477
query63	474	333	329	329
query64	2524	1492	1404	1404
query65	3647	3723	3544	3544
query66	762	369	359	359
query67	16295	15536	15382	15382
query68	8472	648	642	642
query69	569	338	338	338
query70	1955	2038	1960	1960
query71	397	301	312	301
query72	6537	3373	3427	3373
query73	728	334	317	317
query74	6260	5986	5820	5820
query75	4699	3679	3602	3602
query76	4931	1137	1170	1137
query77	689	249	250	249
query78	33651	48293	52804	48293
query79	14819	626	632	626
query80	4275	375	370	370
query81	566	232	228	228
query82	1189	95	91	91
query83	334	132	132	132
query84	251	67	69	67
query85	1710	288	271	271
query86	458	360	425	360
query87	3264	3035	3037	3035
query88	6769	2343	2314	2314
query89	407	301	289	289
query90	2498	209	210	209
query91	167	117	114	114
query92	64	50	51	50
query93	4698	592	568	568
query94	1662	204	203	203
query95	1110	1064	1036	1036
query96	649	342	343	342
query97	6506	6366	6221	6221
query98	184	176	172	172
query99	3666	933	850	850
Total cold run time: 345140 ms
Total hot run time: 239459 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.48 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 32ce302111af3695d834a1142fa03881d03bee02, data reload: false

query1	0.03	0.02	0.03
query2	0.06	0.02	0.02
query3	0.24	0.06	0.04
query4	1.83	0.06	0.06
query5	0.52	0.52	0.52
query6	1.24	0.68	0.62
query7	0.01	0.01	0.00
query8	0.03	0.02	0.02
query9	0.52	0.47	0.49
query10	0.55	0.52	0.53
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.62	0.62	0.62
query14	0.79	0.78	0.79
query15	0.79	0.78	0.78
query16	0.36	0.39	0.39
query17	1.02	1.00	1.02
query18	0.24	0.26	0.26
query19	1.92	1.87	1.86
query20	0.01	0.01	0.01
query21	15.47	0.59	0.56
query22	2.14	2.33	1.26
query23	17.30	0.93	0.84
query24	7.52	1.42	0.93
query25	1.54	0.11	0.11
query26	0.30	0.15	0.13
query27	0.10	0.12	0.13
query28	6.23	0.75	0.71
query29	12.80	2.29	2.46
query30	0.67	0.61	0.57
query31	2.83	0.36	0.38
query32	3.43	0.50	0.49
query33	3.07	3.08	3.10
query34	15.27	4.82	4.78
query35	4.86	4.83	4.85
query36	1.06	1.01	1.01
query37	0.05	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.01	0.01
query40	0.15	0.14	0.14
query41	0.06	0.01	0.01
query42	0.02	0.01	0.01
query43	0.02	0.01	0.02
Total cold run time: 105.95 s
Total hot run time: 30.48 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 32ce302111af3695d834a1142fa03881d03bee02 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

@xiaokang xiaokang closed this Feb 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants