Skip to content

Conversation

@AshinGau
Copy link
Member

@AshinGau AshinGau commented Jun 26, 2024

Proposed changes

Add session variable fetch_splits_max_wait_time to control the max wait time of getting splits to prevent long waiting time.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@AshinGau
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39774 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8fb6f48efe6ac347d1cc3a041c07aedde38933ad, data reload: false

------ Round 1 ----------------------------------
q1	18000	4636	4390	4390
q2	2348	187	184	184
q3	10603	1100	1037	1037
q4	10210	817	760	760
q5	7452	2785	2614	2614
q6	214	131	134	131
q7	939	600	623	600
q8	9241	2057	2081	2057
q9	9002	6532	6471	6471
q10	9040	3749	3693	3693
q11	441	240	246	240
q12	422	230	231	230
q13	18342	2985	3000	2985
q14	251	223	215	215
q15	519	483	460	460
q16	505	379	377	377
q17	965	675	738	675
q18	7997	7411	7495	7411
q19	4645	1432	1449	1432
q20	655	329	323	323
q21	4991	3144	3210	3144
q22	380	368	345	345
Total cold run time: 117162 ms
Total hot run time: 39774 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4471	4257	4265	4257
q2	393	270	272	270
q3	2959	2793	2753	2753
q4	1842	1667	1584	1584
q5	5279	5269	5262	5262
q6	216	128	127	127
q7	2125	1757	1751	1751
q8	3193	3311	3307	3307
q9	8356	8321	8405	8321
q10	3913	3657	3609	3609
q11	576	495	497	495
q12	777	613	570	570
q13	16285	2982	3013	2982
q14	286	275	269	269
q15	522	476	484	476
q16	468	435	409	409
q17	1786	1486	1460	1460
q18	7698	7601	7339	7339
q19	1675	1420	1533	1420
q20	1985	1772	1791	1772
q21	4765	4751	4736	4736
q22	624	532	558	532
Total cold run time: 70194 ms
Total hot run time: 53701 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174910 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8fb6f48efe6ac347d1cc3a041c07aedde38933ad, data reload: false

query1	915	375	372	372
query2	6428	2412	2335	2335
query3	6642	212	212	212
query4	19122	17158	17342	17158
query5	4218	486	482	482
query6	257	167	164	164
query7	4592	298	309	298
query8	318	281	287	281
query9	8558	2469	2446	2446
query10	609	314	279	279
query11	10425	9989	10012	9989
query12	132	83	84	83
query13	1636	393	391	391
query14	10304	8141	8166	8141
query15	293	197	200	197
query16	8084	280	277	277
query17	1890	548	550	548
query18	2008	288	280	280
query19	214	163	163	163
query20	94	82	86	82
query21	206	138	132	132
query22	4338	4035	3916	3916
query23	33808	33063	33120	33063
query24	11973	2787	2831	2787
query25	626	388	392	388
query26	1637	157	158	157
query27	2875	320	329	320
query28	7428	2118	2102	2102
query29	979	656	641	641
query30	279	159	157	157
query31	979	747	745	745
query32	94	57	57	57
query33	790	308	300	300
query34	914	484	484	484
query35	763	675	670	670
query36	1111	939	970	939
query37	163	75	74	74
query38	2892	2781	2733	2733
query39	857	830	796	796
query40	293	135	130	130
query41	55	52	59	52
query42	130	105	103	103
query43	555	543	559	543
query44	1166	742	741	741
query45	199	175	171	171
query46	1086	698	720	698
query47	1814	1789	1741	1741
query48	379	309	303	303
query49	1148	431	424	424
query50	791	409	398	398
query51	6793	6760	6688	6688
query52	106	101	94	94
query53	381	304	306	304
query54	865	468	468	468
query55	76	76	76	76
query56	314	283	286	283
query57	1146	1049	1034	1034
query58	263	265	262	262
query59	3342	3261	3151	3151
query60	316	298	309	298
query61	114	112	109	109
query62	762	439	442	439
query63	330	306	301	301
query64	9878	2251	1751	1751
query65	3212	3136	3152	3136
query66	1351	345	341	341
query67	15953	14995	14942	14942
query68	9627	549	560	549
query69	754	520	418	418
query70	1409	1133	1136	1133
query71	519	281	282	281
query72	8853	5756	5678	5678
query73	2252	348	333	333
query74	5814	5515	5481	5481
query75	5838	2708	2657	2657
query76	5438	922	948	922
query77	759	310	304	304
query78	10369	9996	9703	9703
query79	8408	528	526	526
query80	921	513	474	474
query81	553	230	224	224
query82	236	107	106	106
query83	339	175	169	169
query84	271	90	93	90
query85	951	302	269	269
query86	348	317	336	317
query87	3314	3107	3116	3107
query88	4814	2473	2455	2455
query89	538	398	400	398
query90	1903	193	193	193
query91	129	102	137	102
query92	68	50	50	50
query93	5846	508	501	501
query94	1249	190	188	188
query95	409	324	325	324
query96	618	276	271	271
query97	3204	3028	3041	3028
query98	218	199	198	198
query99	1163	841	858	841
Total cold run time: 299401 ms
Total hot run time: 174910 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.79 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8fb6f48efe6ac347d1cc3a041c07aedde38933ad, data reload: false

query1	0.03	0.03	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.04
query4	1.68	0.09	0.09
query5	0.49	0.47	0.49
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.06	0.04	0.04
query9	0.54	0.47	0.50
query10	0.52	0.52	0.55
query11	0.16	0.12	0.11
query12	0.15	0.12	0.13
query13	0.58	0.57	0.60
query14	0.79	0.77	0.79
query15	0.84	0.82	0.81
query16	0.36	0.36	0.37
query17	0.97	1.00	0.98
query18	0.20	0.25	0.25
query19	1.79	1.68	1.67
query20	0.02	0.01	0.01
query21	15.44	0.65	0.66
query22	3.83	7.41	2.15
query23	18.31	1.43	1.31
query24	2.07	0.22	0.23
query25	0.14	0.09	0.09
query26	0.27	0.18	0.17
query27	0.08	0.08	0.08
query28	13.25	1.03	1.00
query29	12.62	3.27	3.28
query30	0.26	0.06	0.06
query31	2.86	0.37	0.39
query32	3.28	0.48	0.48
query33	2.89	2.92	2.93
query34	17.13	4.40	4.46
query35	4.48	4.50	4.54
query36	0.65	0.47	0.46
query37	0.18	0.16	0.15
query38	0.16	0.14	0.15
query39	0.05	0.04	0.04
query40	0.18	0.17	0.13
query41	0.08	0.05	0.04
query42	0.06	0.04	0.04
query43	0.05	0.04	0.05
Total cold run time: 108.95 s
Total hot run time: 30.79 s

@AshinGau
Copy link
Member Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39727 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f0c1d80f736a2675f5e9a6eacdbb1a0c0e1a02b4, data reload: false

------ Round 1 ----------------------------------
q1	17629	4327	4276	4276
q2	2021	197	199	197
q3	10450	1151	1149	1149
q4	10191	856	770	770
q5	7489	2648	2638	2638
q6	217	134	135	134
q7	943	595	602	595
q8	9222	2076	2046	2046
q9	9045	6490	6439	6439
q10	8952	3664	3689	3664
q11	469	231	236	231
q12	440	230	234	230
q13	18852	2985	2996	2985
q14	267	218	222	218
q15	521	469	476	469
q16	510	371	371	371
q17	958	695	670	670
q18	7980	7391	7350	7350
q19	5907	1499	1576	1499
q20	649	309	322	309
q21	4831	3147	3921	3147
q22	408	340	340	340
Total cold run time: 117951 ms
Total hot run time: 39727 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4431	4225	4237	4225
q2	365	263	254	254
q3	2911	2884	2916	2884
q4	2071	1720	1750	1720
q5	5571	5480	5472	5472
q6	216	128	128	128
q7	2242	1851	1825	1825
q8	3270	3366	3455	3366
q9	8732	8686	8783	8686
q10	4131	3820	3749	3749
q11	576	488	498	488
q12	834	635	651	635
q13	16212	3162	3145	3145
q14	308	264	272	264
q15	541	509	506	506
q16	500	416	442	416
q17	1833	1497	1492	1492
q18	8036	7958	7813	7813
q19	1798	1488	1655	1488
q20	3059	1910	1838	1838
q21	5169	4982	4890	4890
q22	600	562	572	562
Total cold run time: 73406 ms
Total hot run time: 55846 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 175568 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f0c1d80f736a2675f5e9a6eacdbb1a0c0e1a02b4, data reload: false

query1	913	386	379	379
query2	6467	2481	2520	2481
query3	6640	206	217	206
query4	18913	17724	17505	17505
query5	3625	484	473	473
query6	252	183	162	162
query7	4589	296	293	293
query8	307	293	285	285
query9	8620	2474	2448	2448
query10	568	306	289	289
query11	10522	10091	10196	10091
query12	119	90	89	89
query13	1655	369	369	369
query14	10178	7972	7577	7577
query15	233	191	192	191
query16	7813	284	278	278
query17	1870	588	531	531
query18	1991	279	279	279
query19	196	159	158	158
query20	89	87	85	85
query21	216	135	127	127
query22	4564	4446	4262	4262
query23	33989	33555	33892	33555
query24	11155	2882	2910	2882
query25	648	407	400	400
query26	1104	159	160	159
query27	2922	335	331	331
query28	7193	2145	2139	2139
query29	903	650	655	650
query30	264	163	166	163
query31	965	801	767	767
query32	98	55	55	55
query33	766	314	304	304
query34	976	472	487	472
query35	753	645	650	645
query36	1144	984	1005	984
query37	181	77	77	77
query38	2971	2847	2881	2847
query39	877	858	815	815
query40	205	133	132	132
query41	60	53	56	53
query42	110	104	107	104
query43	595	564	575	564
query44	1204	733	723	723
query45	197	167	163	163
query46	1085	752	729	729
query47	1885	1790	1787	1787
query48	385	305	302	302
query49	871	422	516	422
query50	763	386	402	386
query51	6924	6760	6759	6759
query52	99	95	90	90
query53	359	288	300	288
query54	868	450	440	440
query55	73	74	72	72
query56	283	272	254	254
query57	1132	1057	1038	1038
query58	255	235	283	235
query59	3495	3167	3310	3167
query60	322	278	302	278
query61	95	110	89	89
query62	598	432	428	428
query63	319	293	289	289
query64	8837	2249	1776	1776
query65	3219	3115	3109	3109
query66	804	321	326	321
query67	15758	15055	14854	14854
query68	8789	555	549	549
query69	769	548	482	482
query70	1395	1179	1082	1082
query71	541	283	275	275
query72	8895	5361	5479	5361
query73	2215	324	328	324
query74	5893	5537	5546	5537
query75	5476	2626	2653	2626
query76	5559	970	952	952
query77	758	295	298	295
query78	10586	10034	9790	9790
query79	7528	505	504	504
query80	870	470	461	461
query81	570	217	224	217
query82	261	103	105	103
query83	349	171	165	165
query84	319	85	83	83
query85	952	278	266	266
query86	339	313	321	313
query87	3266	3110	3096	3096
query88	4375	2458	2421	2421
query89	513	370	388	370
query90	1970	190	187	187
query91	127	101	96	96
query92	66	48	52	48
query93	5922	500	509	500
query94	1270	185	185	185
query95	412	318	325	318
query96	620	308	266	266
query97	3225	3033	3004	3004
query98	218	194	199	194
query99	1118	870	843	843
Total cold run time: 293215 ms
Total hot run time: 175568 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.85 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f0c1d80f736a2675f5e9a6eacdbb1a0c0e1a02b4, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.08
query5	0.51	0.48	0.49
query6	1.17	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.56	0.48	0.49
query10	0.54	0.54	0.55
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.61
query14	0.76	0.78	0.78
query15	0.82	0.82	0.82
query16	0.37	0.34	0.34
query17	1.00	1.01	1.02
query18	0.22	0.24	0.24
query19	1.74	1.71	1.66
query20	0.01	0.00	0.00
query21	15.44	0.75	0.67
query22	4.03	7.05	2.25
query23	18.31	1.44	1.26
query24	2.14	0.23	0.22
query25	0.15	0.10	0.08
query26	0.26	0.18	0.17
query27	0.08	0.08	0.08
query28	13.24	1.02	1.00
query29	12.61	3.25	3.25
query30	0.25	0.06	0.06
query31	2.86	0.39	0.38
query32	3.30	0.48	0.48
query33	2.84	2.90	2.92
query34	17.16	4.43	4.41
query35	4.48	4.53	4.47
query36	0.65	0.47	0.46
query37	0.18	0.15	0.15
query38	0.16	0.14	0.14
query39	0.04	0.04	0.03
query40	0.19	0.14	0.14
query41	0.08	0.04	0.04
query42	0.05	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.23 s
Total hot run time: 30.85 s

@AshinGau
Copy link
Member Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39890 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fcb6895e6071aa22fb2397a3372c0fb7deefd755, data reload: false

------ Round 1 ----------------------------------
q1	17602	4316	4308	4308
q2	2020	200	194	194
q3	10447	1243	1227	1227
q4	10186	747	790	747
q5	7498	2658	2618	2618
q6	224	136	137	136
q7	959	588	611	588
q8	9223	2075	2047	2047
q9	8795	6498	6476	6476
q10	9044	3775	3751	3751
q11	454	235	232	232
q12	463	229	217	217
q13	17774	2983	2966	2966
q14	274	213	222	213
q15	513	484	472	472
q16	520	377	373	373
q17	946	658	614	614
q18	7932	7426	7352	7352
q19	5374	1491	1448	1448
q20	658	316	321	316
q21	4895	3256	3944	3256
q22	411	344	339	339
Total cold run time: 116212 ms
Total hot run time: 39890 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4388	4281	4281	4281
q2	382	271	279	271
q3	3005	2853	2875	2853
q4	1993	1690	1664	1664
q5	5666	5535	5487	5487
q6	225	135	132	132
q7	2235	1878	1852	1852
q8	3286	3413	3447	3413
q9	8663	8668	8833	8668
q10	4133	3965	3687	3687
q11	612	498	492	492
q12	841	638	666	638
q13	16253	3214	3162	3162
q14	314	271	285	271
q15	540	487	481	481
q16	487	443	428	428
q17	1801	1529	1480	1480
q18	8095	8152	7905	7905
q19	1823	1677	1597	1597
q20	2911	1892	1847	1847
q21	5188	4929	4737	4737
q22	656	548	571	548
Total cold run time: 73497 ms
Total hot run time: 55894 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174986 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fcb6895e6071aa22fb2397a3372c0fb7deefd755, data reload: false

query1	931	384	371	371
query2	6464	2326	2311	2311
query3	6632	204	211	204
query4	21528	17344	17314	17314
query5	3693	494	479	479
query6	272	186	163	163
query7	4588	303	287	287
query8	317	284	301	284
query9	8547	2371	2363	2363
query10	555	305	304	304
query11	10503	10088	10040	10040
query12	112	85	82	82
query13	1630	365	372	365
query14	10098	7541	7663	7541
query15	233	189	191	189
query16	7633	274	259	259
query17	1405	536	535	535
query18	1915	273	267	267
query19	190	149	149	149
query20	89	82	81	81
query21	208	136	129	129
query22	4254	4099	3964	3964
query23	34012	33806	33653	33653
query24	11002	2900	2880	2880
query25	604	385	372	372
query26	721	158	156	156
query27	2302	332	324	324
query28	5761	2132	2126	2126
query29	911	633	639	633
query30	239	152	156	152
query31	1000	769	758	758
query32	100	62	59	59
query33	660	311	288	288
query34	890	473	504	473
query35	738	671	641	641
query36	1138	982	982	982
query37	143	101	83	83
query38	2947	2832	2862	2832
query39	915	852	819	819
query40	207	139	128	128
query41	57	50	51	50
query42	114	104	107	104
query43	621	576	558	558
query44	1078	753	725	725
query45	193	167	172	167
query46	1071	700	716	700
query47	1800	1787	1765	1765
query48	372	302	303	302
query49	858	444	422	422
query50	770	385	385	385
query51	6860	6821	6774	6774
query52	102	101	88	88
query53	369	298	296	296
query54	882	448	439	439
query55	75	72	73	72
query56	290	268	288	268
query57	1133	1029	1059	1029
query58	269	273	264	264
query59	3399	3177	3103	3103
query60	297	289	290	289
query61	92	88	90	88
query62	625	464	474	464
query63	325	295	290	290
query64	8506	2225	1756	1756
query65	3174	3103	3111	3103
query66	759	328	341	328
query67	15404	15172	15066	15066
query68	4575	537	537	537
query69	638	434	339	339
query70	1167	1156	1190	1156
query71	407	296	288	288
query72	7322	5762	5604	5604
query73	760	326	325	325
query74	5893	5464	5460	5460
query75	3391	2707	2680	2680
query76	2463	1021	938	938
query77	633	306	312	306
query78	10602	9979	9773	9773
query79	2883	528	517	517
query80	1991	483	501	483
query81	555	224	221	221
query82	780	110	111	110
query83	336	236	173	173
query84	270	94	85	85
query85	1671	282	288	282
query86	473	306	317	306
query87	3278	3060	3088	3060
query88	4347	2391	2363	2363
query89	495	385	403	385
query90	1841	192	196	192
query91	131	98	100	98
query92	64	49	49	49
query93	4328	515	503	503
query94	1215	190	185	185
query95	409	314	322	314
query96	618	274	273	273
query97	3281	3019	3030	3019
query98	218	199	193	193
query99	1168	850	834	834
Total cold run time: 273785 ms
Total hot run time: 174986 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.5 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fcb6895e6071aa22fb2397a3372c0fb7deefd755, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.04
query3	0.22	0.05	0.06
query4	1.67	0.07	0.08
query5	0.56	0.49	0.49
query6	1.15	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.56	0.50	0.50
query10	0.55	0.53	0.54
query11	0.14	0.11	0.11
query12	0.15	0.11	0.12
query13	0.59	0.59	0.61
query14	0.80	0.77	0.80
query15	0.83	0.82	0.81
query16	0.37	0.35	0.34
query17	1.04	1.04	1.02
query18	0.22	0.26	0.25
query19	1.84	1.72	1.71
query20	0.01	0.00	0.01
query21	15.44	0.77	0.66
query22	4.15	7.43	1.90
query23	18.28	1.31	1.18
query24	2.10	0.22	0.21
query25	0.13	0.09	0.08
query26	0.26	0.17	0.18
query27	0.08	0.08	0.08
query28	13.29	1.03	0.99
query29	12.61	3.30	3.26
query30	0.25	0.06	0.05
query31	2.87	0.39	0.40
query32	3.28	0.47	0.49
query33	2.96	2.96	2.93
query34	17.02	4.40	4.39
query35	4.44	4.49	4.53
query36	0.65	0.47	0.52
query37	0.19	0.16	0.16
query38	0.16	0.15	0.14
query39	0.04	0.04	0.03
query40	0.20	0.14	0.15
query41	0.10	0.04	0.04
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.47 s
Total hot run time: 30.5 s

Copy link
Contributor

@wsjz wsjz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Jul 1, 2024

PR approved by anyone and no changes requested.

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 1, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jul 1, 2024

PR approved by at least one committer and no changes requested.

@morningman morningman merged commit fc65bc4 into apache:master Jul 1, 2024
dataroaring pushed a commit that referenced this pull request Jul 2, 2024
Add session variable `fetch_splits_max_wait_time` to control the max
wait time of getting splits to prevent long waiting time.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.0-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants