[opt](split) generate and get split batch concurrently #36045

AshinGau · 2024-06-07T08:22:48Z

Proposed changes

Generate and get split batch concurrently.
SplitSource.getNextBatch remove the synchronization, and make each get their splits concurrently, and SplitAssignment generates splits asynchronously.

Because obtaining partitions and splits is not stable, once encountering unstable situations, the time consumption will increase several times. The batch interface will retrieve split data in batches, which greatly increases the number of unstable situations encountered. So temporarily close the batch interface.

Improvements

Query a big table with 60591 partitions and 1209172 splits:

|   0:VHIVE_SCAN_NODE(136)                                                 |
|      table: level3partition                                              |
|      inputSplitNum=1209172, totalFileSize=6527616577, scanRanges=1209172 |
|      partition=60591/60591                                               |
|      cardinality=1, numNodes=3                                           |
|      pushdown agg=COUNT                                                  |

set max_external_file_cache_num=0 in fe/fe.conf to turn off the cache;
set num_partitions_in_batch_mode=-1 in session variables to turn off the batch mode.

Batch Mode	With Cache	Before Opt.	After Opt.
No	No	393.33s	123.25s
No	Yes	56.71s(480s First Run)	49.30s(124s First Run)
Yes	No	-	46.32s
Yes	Yes	-	29.73s

doris-robot · 2024-06-07T08:22:53Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions · 2024-06-07T08:29:34Z

clang-tidy review says "All clean, LGTM! 👍"

AshinGau · 2024-06-07T08:41:07Z

run buildall

github-actions · 2024-06-07T08:46:16Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-06-07T09:11:44Z

TeamCity be ut coverage result:
Function Coverage: 36.44% (8975/24632)
Line Coverage: 27.97% (73335/262204)
Region Coverage: 27.37% (38009/138880)
Branch Coverage: 23.99% (19302/80462)
Coverage Report: http://coverage.selectdb-in.cc/coverage/444772611312cb8d06d921cbcd2ad42886fef786_444772611312cb8d06d921cbcd2ad42886fef786/report/index.html

doris-robot · 2024-06-07T09:20:02Z

TPC-H: Total hot run time: 41508 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 444772611312cb8d06d921cbcd2ad42886fef786, data reload: false

------ Round 1 ----------------------------------
q1	17608	4312	4273	4273
q2	2019	196	193	193
q3	10450	1229	1103	1103
q4	10209	734	886	734
q5	7479	2698	2668	2668
q6	222	135	135	135
q7	962	614	610	610
q8	9240	2138	2124	2124
q9	9398	6712	6687	6687
q10	9716	3887	3847	3847
q11	433	246	237	237
q12	460	231	234	231
q13	18513	3212	3155	3155
q14	281	215	213	213
q15	507	455	473	455
q16	510	389	393	389
q17	978	648	639	639
q18	8469	7881	7807	7807
q19	5523	1412	1406	1406
q20	637	325	329	325
q21	5139	4082	3944	3944
q22	403	359	333	333
Total cold run time: 119156 ms
Total hot run time: 41508 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4528	4433	4391	4391
q2	373	253	261	253
q3	3140	3044	2823	2823
q4	1874	1608	1654	1608
q5	5474	5500	5476	5476
q6	223	129	125	125
q7	2156	1814	1774	1774
q8	3235	3368	3323	3323
q9	8550	8674	8655	8655
q10	4046	3805	3605	3605
q11	565	474	511	474
q12	780	626	632	626
q13	17050	3110	3179	3110
q14	290	290	256	256
q15	522	491	488	488
q16	471	422	433	422
q17	1785	1488	1514	1488
q18	8003	7371	7437	7371
q19	3605	1598	1508	1508
q20	2092	1784	1770	1770
q21	4902	4708	4623	4623
q22	627	563	567	563
Total cold run time: 74291 ms
Total hot run time: 54732 ms

doris-robot · 2024-06-07T09:31:01Z

TPC-DS: Total hot run time: 172672 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 444772611312cb8d06d921cbcd2ad42886fef786, data reload: false

query1	932	385	372	372
query2	6438	2370	2310	2310
query3	6648	208	212	208
query4	19745	17598	17427	17427
query5	4132	451	456	451
query6	246	169	155	155
query7	4600	293	290	290
query8	312	306	310	306
query9	8499	2396	2376	2376
query10	458	299	282	282
query11	10581	10129	10151	10129
query12	146	85	91	85
query13	1652	380	382	380
query14	10212	7578	6940	6940
query15	231	189	185	185
query16	7825	269	273	269
query17	1817	539	527	527
query18	1951	283	283	283
query19	198	159	164	159
query20	92	89	84	84
query21	251	135	132	132
query22	4524	3988	3978	3978
query23	33932	33069	33011	33011
query24	11090	2893	2915	2893
query25	640	384	363	363
query26	1375	155	155	155
query27	2946	316	319	316
query28	7582	2030	2027	2027
query29	907	599	604	599
query30	291	147	148	147
query31	945	714	726	714
query32	90	53	56	53
query33	750	270	269	269
query34	976	478	461	461
query35	753	618	615	615
query36	1121	936	942	936
query37	150	66	70	66
query38	2850	2711	2704	2704
query39	862	806	795	795
query40	206	130	126	126
query41	56	52	54	52
query42	117	97	98	97
query43	589	573	574	573
query44	1273	727	751	727
query45	222	160	164	160
query46	1069	710	708	708
query47	1863	1770	1811	1770
query48	374	288	290	288
query49	1081	399	435	399
query50	772	402	392	392
query51	6783	6694	6572	6572
query52	103	98	94	94
query53	363	295	290	290
query54	869	439	445	439
query55	73	74	71	71
query56	292	253	262	253
query57	1161	1067	1056	1056
query58	242	239	243	239
query59	3558	3171	3218	3171
query60	293	295	279	279
query61	87	86	86	86
query62	642	453	461	453
query63	317	292	299	292
query64	8910	2231	1728	1728
query65	3184	3125	3117	3117
query66	848	334	344	334
query67	15548	14921	14959	14921
query68	4612	551	544	544
query69	524	414	414	414
query70	1184	1120	1085	1085
query71	434	264	276	264
query72	7573	5675	5309	5309
query73	739	323	320	320
query74	5892	5533	5411	5411
query75	3497	2675	2647	2647
query76	2857	931	928	928
query77	493	293	332	293
query78	10507	9849	9800	9800
query79	2467	511	519	511
query80	2090	451	456	451
query81	600	217	221	217
query82	961	103	102	102
query83	329	175	169	169
query84	273	85	86	85
query85	1880	283	264	264
query86	497	337	318	318
query87	3235	3095	3065	3065
query88	3930	2344	2336	2336
query89	474	403	393	393
query90	1808	191	188	188
query91	125	93	145	93
query92	66	48	48	48
query93	2721	512	497	497
query94	1217	184	185	184
query95	399	307	314	307
query96	588	267	262	262
query97	3163	2997	3006	2997
query98	240	198	197	197
query99	1308	834	836	834
Total cold run time: 276328 ms
Total hot run time: 172672 ms

doris-robot · 2024-06-07T09:36:33Z

ClickBench: Total hot run time: 30.64 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 444772611312cb8d06d921cbcd2ad42886fef786, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.07
query4	1.66	0.07	0.08
query5	0.50	0.48	0.50
query6	1.12	0.72	0.72
query7	0.02	0.02	0.01
query8	0.06	0.04	0.04
query9	0.55	0.49	0.49
query10	0.55	0.55	0.56
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.58	0.60
query14	0.79	0.76	0.80
query15	0.84	0.82	0.82
query16	0.36	0.37	0.37
query17	0.97	0.96	1.00
query18	0.20	0.25	0.24
query19	1.81	1.84	1.71
query20	0.01	0.01	0.01
query21	15.56	0.65	0.65
query22	4.38	7.32	2.06
query23	18.31	1.39	1.22
query24	2.13	0.22	0.21
query25	0.15	0.09	0.09
query26	0.26	0.18	0.18
query27	0.08	0.08	0.07
query28	13.23	1.02	1.00
query29	13.29	3.33	3.28
query30	0.24	0.06	0.05
query31	2.87	0.40	0.39
query32	3.27	0.47	0.47
query33	2.87	2.89	2.92
query34	17.05	4.44	4.39
query35	4.53	4.50	4.51
query36	0.72	0.49	0.47
query37	0.17	0.14	0.14
query38	0.15	0.15	0.14
query39	0.04	0.04	0.03
query40	0.18	0.14	0.14
query41	0.09	0.04	0.04
query42	0.05	0.04	0.04
query43	0.04	0.04	0.03
Total cold run time: 110.33 s
Total hot run time: 30.64 s

github-actions · 2024-06-17T02:40:50Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-06-17T02:49:51Z

clang-tidy review says "All clean, LGTM! 👍"

AshinGau · 2024-06-17T03:12:07Z

run buildall

doris-robot · 2024-06-17T03:55:53Z

TeamCity be ut coverage result:
Function Coverage: 36.47% (9002/24680)
Line Coverage: 28.01% (73736/263215)
Region Coverage: 27.49% (38301/139349)
Branch Coverage: 24.18% (19523/80734)
Coverage Report: http://coverage.selectdb-in.cc/coverage/dc6b42a899b382ffa47dba50c31d0735d8d94539_dc6b42a899b382ffa47dba50c31d0735d8d94539/report/index.html

doris-robot · 2024-06-17T04:01:05Z

TPC-H: Total hot run time: 39650 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dc6b42a899b382ffa47dba50c31d0735d8d94539, data reload: false

------ Round 1 ----------------------------------
q1	17609	5042	4351	4351
q2	2011	207	192	192
q3	10504	1209	1070	1070
q4	10194	725	754	725
q5	7514	2706	2611	2611
q6	221	138	137	137
q7	958	616	597	597
q8	9217	2081	2064	2064
q9	8950	6470	6454	6454
q10	9039	3717	3685	3685
q11	452	242	240	240
q12	544	232	230	230
q13	17756	3008	2979	2979
q14	257	215	215	215
q15	509	472	473	472
q16	526	383	378	378
q17	959	646	717	646
q18	8138	7474	7342	7342
q19	5603	1494	1410	1410
q20	656	337	324	324
q21	5162	3890	3188	3188
q22	394	346	340	340
Total cold run time: 117173 ms
Total hot run time: 39650 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4407	4197	4284	4197
q2	368	275	277	275
q3	2982	3025	2918	2918
q4	1988	1756	1768	1756
q5	5609	5517	5482	5482
q6	226	132	140	132
q7	2236	1957	1828	1828
q8	3273	3384	3429	3384
q9	8747	8765	8818	8765
q10	4099	3772	3817	3772
q11	605	505	496	496
q12	810	623	685	623
q13	16184	3183	3194	3183
q14	319	292	261	261
q15	534	487	491	487
q16	515	449	445	445
q17	1843	1522	1500	1500
q18	8126	7987	7767	7767
q19	1810	1576	1549	1549
q20	2125	1878	1848	1848
q21	7582	4848	4784	4784
q22	647	588	580	580
Total cold run time: 75035 ms
Total hot run time: 56032 ms

doris-robot · 2024-06-17T04:12:15Z

TPC-DS: Total hot run time: 173388 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dc6b42a899b382ffa47dba50c31d0735d8d94539, data reload: false

query1	921	386	370	370
query2	6452	2541	2215	2215
query3	6632	213	205	205
query4	19934	17362	17395	17362
query5	3539	472	464	464
query6	239	154	159	154
query7	4580	299	293	293
query8	328	285	291	285
query9	8666	2364	2351	2351
query10	564	300	273	273
query11	10455	9875	10016	9875
query12	134	86	86	86
query13	1637	351	370	351
query14	10117	6986	7051	6986
query15	228	196	185	185
query16	7753	279	273	273
query17	1770	540	509	509
query18	1963	276	268	268
query19	199	154	150	150
query20	91	84	82	82
query21	211	125	123	123
query22	4350	4080	3945	3945
query23	33841	33732	33678	33678
query24	11088	2947	2804	2804
query25	579	374	383	374
query26	718	155	155	155
query27	2367	330	317	317
query28	5908	2121	2089	2089
query29	934	621	643	621
query30	233	152	155	152
query31	959	786	750	750
query32	92	54	54	54
query33	702	283	279	279
query34	958	492	469	469
query35	749	630	650	630
query36	1113	992	974	974
query37	174	84	68	68
query38	2926	2793	2838	2793
query39	913	849	841	841
query40	204	125	125	125
query41	52	59	53	53
query42	110	102	101	101
query43	599	565	552	552
query44	1161	715	715	715
query45	197	164	167	164
query46	1070	700	708	700
query47	1893	1738	1763	1738
query48	375	300	293	293
query49	839	403	402	402
query50	773	391	380	380
query51	6787	6588	6594	6588
query52	102	92	91	91
query53	355	288	289	288
query54	863	447	455	447
query55	81	73	74	73
query56	287	279	279	279
query57	1136	1021	1026	1021
query58	260	241	245	241
query59	3373	3054	3243	3054
query60	321	282	279	279
query61	111	113	107	107
query62	606	437	466	437
query63	327	289	289	289
query64	8551	2315	1830	1830
query65	3200	3100	3155	3100
query66	752	338	332	332
query67	15432	15135	14999	14999
query68	4612	536	528	528
query69	543	335	312	312
query70	1119	1134	1100	1100
query71	403	282	273	273
query72	8076	5843	5585	5585
query73	756	324	322	322
query74	6046	5472	5584	5472
query75	3567	2648	2702	2648
query76	2260	959	883	883
query77	646	291	287	287
query78	10477	9860	9871	9860
query79	8239	519	517	517
query80	1813	466	459	459
query81	601	217	222	217
query82	1670	104	98	98
query83	346	162	160	160
query84	278	83	85	83
query85	2024	350	273	273
query86	479	308	312	308
query87	3278	3076	3080	3076
query88	5447	2363	2343	2343
query89	499	382	391	382
query90	1811	191	186	186
query91	124	99	97	97
query92	61	48	49	48
query93	6406	507	500	500
query94	1178	190	181	181
query95	402	322	327	322
query96	625	262	261	261
query97	3266	3059	3120	3059
query98	225	206	191	191
query99	1247	829	826	826
Total cold run time: 283358 ms
Total hot run time: 173388 ms

doris-robot · 2024-06-17T04:17:47Z

ClickBench: Total hot run time: 30.69 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dc6b42a899b382ffa47dba50c31d0735d8d94539, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.66	0.07	0.08
query5	0.52	0.47	0.49
query6	1.12	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.04
query9	0.54	0.49	0.49
query10	0.54	0.54	0.55
query11	0.16	0.11	0.11
query12	0.15	0.12	0.11
query13	0.59	0.59	0.60
query14	0.78	0.78	0.78
query15	0.83	0.81	0.83
query16	0.37	0.38	0.38
query17	1.05	0.99	1.03
query18	0.20	0.26	0.25
query19	1.80	1.69	1.71
query20	0.02	0.01	0.01
query21	15.43	0.65	0.64
query22	4.56	6.92	1.95
query23	18.30	1.39	1.37
query24	2.14	0.22	0.21
query25	0.15	0.08	0.08
query26	0.27	0.18	0.18
query27	0.08	0.08	0.08
query28	13.26	1.04	1.01
query29	12.61	3.28	3.23
query30	0.25	0.08	0.07
query31	2.85	0.39	0.38
query32	3.27	0.48	0.47
query33	2.94	2.91	2.96
query34	17.18	4.42	4.42
query35	4.47	4.50	4.43
query36	0.64	0.46	0.47
query37	0.19	0.15	0.15
query38	0.16	0.15	0.15
query39	0.04	0.03	0.03
query40	0.18	0.15	0.15
query41	0.10	0.05	0.04
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.93 s
Total hot run time: 30.69 s

morningman

LGTM

github-actions · 2024-06-17T07:06:09Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-06-17T07:06:11Z

PR approved by anyone and no changes requested.

AshinGau · 2024-06-18T02:53:06Z

run buildall

github-actions · 2024-06-18T02:54:35Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-06-18T02:57:39Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-06-18T03:25:13Z

TeamCity be ut coverage result:
Function Coverage: 36.48% (9003/24681)
Line Coverage: 28.01% (73730/263205)
Region Coverage: 27.48% (38292/139347)
Branch Coverage: 24.17% (19513/80734)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5e1e3c6ff6ffca644ab0a4c202e13fde7213119a_5e1e3c6ff6ffca644ab0a4c202e13fde7213119a/report/index.html

morningman · 2024-06-18T08:05:10Z

run buildall

doris-robot · 2024-06-18T08:28:46Z

TeamCity be ut coverage result:
Function Coverage: 36.49% (9005/24681)
Line Coverage: 28.03% (73762/263189)
Region Coverage: 27.50% (38311/139338)
Branch Coverage: 24.19% (19524/80704)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5e1e3c6ff6ffca644ab0a4c202e13fde7213119a_5e1e3c6ff6ffca644ab0a4c202e13fde7213119a/report/index.html

AshinGau · 2024-06-19T01:47:30Z

run buildall

github-actions · 2024-06-19T01:53:26Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-06-19T02:22:59Z

TPC-H: Total hot run time: 39718 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 03123e6500ecd0c0bcb15d8fdf8e67170f3f004c, data reload: false

------ Round 1 ----------------------------------
q1	17613	4378	4231	4231
q2	2019	193	195	193
q3	10439	1196	1104	1104
q4	10195	798	720	720
q5	7469	2682	2649	2649
q6	216	138	133	133
q7	963	611	601	601
q8	9227	2046	2062	2046
q9	8987	6503	6500	6500
q10	8933	3730	3716	3716
q11	450	246	243	243
q12	489	229	222	222
q13	17758	2976	2963	2963
q14	260	218	216	216
q15	526	482	481	481
q16	534	377	369	369
q17	968	679	693	679
q18	7937	7387	7384	7384
q19	8543	1502	1470	1470
q20	665	318	318	318
q21	4846	3134	3905	3134
q22	391	347	346	346
Total cold run time: 119428 ms
Total hot run time: 39718 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4376	4236	4246	4236
q2	370	259	288	259
q3	3093	2905	2913	2905
q4	2006	1758	1711	1711
q5	5503	5469	5456	5456
q6	227	135	128	128
q7	2260	1815	1819	1815
q8	3263	3410	3416	3410
q9	8605	8860	8678	8678
q10	4185	3706	3751	3706
q11	597	493	498	493
q12	816	690	639	639
q13	15840	3119	3168	3119
q14	294	284	258	258
q15	533	484	488	484
q16	470	442	424	424
q17	1820	1517	1490	1490
q18	8192	7921	7795	7795
q19	2575	1656	1670	1656
q20	3228	1882	1815	1815
q21	5078	4911	4696	4696
q22	664	540	542	540
Total cold run time: 73995 ms
Total hot run time: 55713 ms

doris-robot · 2024-06-19T02:25:06Z

TeamCity be ut coverage result:
Function Coverage: 36.47% (9004/24686)
Line Coverage: 28.02% (73760/263266)
Region Coverage: 27.50% (38321/139362)
Branch Coverage: 24.19% (19526/80718)
Coverage Report: http://coverage.selectdb-in.cc/coverage/03123e6500ecd0c0bcb15d8fdf8e67170f3f004c_03123e6500ecd0c0bcb15d8fdf8e67170f3f004c/report/index.html

doris-robot · 2024-06-19T02:33:52Z

TPC-DS: Total hot run time: 174369 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 03123e6500ecd0c0bcb15d8fdf8e67170f3f004c, data reload: false

query1	936	375	380	375
query2	6469	2452	2288	2288
query3	6641	203	208	203
query4	18964	17416	17460	17416
query5	3551	472	441	441
query6	236	159	175	159
query7	4582	301	290	290
query8	351	295	286	286
query9	8574	2410	2386	2386
query10	576	312	281	281
query11	10682	10215	10246	10215
query12	121	86	86	86
query13	1645	370	380	370
query14	9501	7562	7731	7562
query15	231	200	199	199
query16	7752	268	257	257
query17	1905	528	503	503
query18	1919	267	274	267
query19	189	150	150	150
query20	91	83	83	83
query21	216	126	119	119
query22	4606	4465	4157	4157
query23	33712	33806	33650	33650
query24	10625	2890	2840	2840
query25	590	367	365	365
query26	737	157	156	156
query27	2289	310	334	310
query28	6009	2115	2094	2094
query29	852	640	625	625
query30	258	157	167	157
query31	969	788	738	738
query32	95	53	55	53
query33	658	293	313	293
query34	915	476	470	470
query35	721	650	616	616
query36	1139	981	1001	981
query37	139	73	70	70
query38	2930	2848	2818	2818
query39	886	839	817	817
query40	212	124	121	121
query41	51	49	53	49
query42	114	94	93	93
query43	565	542	549	542
query44	1080	714	733	714
query45	192	160	163	160
query46	1064	735	713	713
query47	1890	1810	1809	1809
query48	373	290	289	289
query49	835	392	423	392
query50	763	377	381	377
query51	6790	6729	6637	6637
query52	101	93	86	86
query53	355	286	286	286
query54	847	438	442	438
query55	72	72	74	72
query56	266	248	253	248
query57	1143	1086	1093	1086
query58	264	246	249	246
query59	3432	3226	3290	3226
query60	293	257	264	257
query61	91	92	92	92
query62	602	433	446	433
query63	313	299	287	287
query64	8569	2302	1722	1722
query65	3205	3119	3105	3105
query66	736	325	324	324
query67	15558	15246	14861	14861
query68	4468	526	546	526
query69	546	383	381	381
query70	1162	1176	1130	1130
query71	365	270	260	260
query72	7092	5755	5314	5314
query73	751	317	318	317
query74	5945	5505	5531	5505
query75	3321	2678	2627	2627
query76	2242	953	887	887
query77	455	290	287	287
query78	10373	9772	9809	9772
query79	2517	520	516	516
query80	1083	457	520	457
query81	590	221	219	219
query82	714	97	101	97
query83	255	160	161	160
query84	233	85	83	83
query85	1686	285	263	263
query86	496	293	297	293
query87	3276	3099	3101	3099
query88	3943	2333	2324	2324
query89	503	386	374	374
query90	1745	182	183	182
query91	125	102	98	98
query92	61	50	49	49
query93	2188	501	501	501
query94	1129	181	183	181
query95	392	310	307	307
query96	586	267	272	267
query97	3289	3040	3046	3040
query98	212	200	202	200
query99	1155	842	844	842
Total cold run time: 265936 ms
Total hot run time: 174369 ms

doris-robot · 2024-06-19T02:39:22Z

ClickBench: Total hot run time: 30.16 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 03123e6500ecd0c0bcb15d8fdf8e67170f3f004c, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.08
query5	0.50	0.49	0.48
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.55	0.49	0.50
query10	0.56	0.55	0.53
query11	0.16	0.11	0.11
query12	0.15	0.13	0.12
query13	0.60	0.58	0.60
query14	0.77	0.79	0.76
query15	0.82	0.81	0.81
query16	0.36	0.37	0.37
query17	1.04	1.05	1.04
query18	0.20	0.25	0.23
query19	1.73	1.69	1.75
query20	0.01	0.01	0.01
query21	15.40	0.64	0.64
query22	4.57	7.06	1.45
query23	18.28	1.36	1.34
query24	2.10	0.23	0.22
query25	0.16	0.08	0.09
query26	0.27	0.17	0.17
query27	0.09	0.07	0.08
query28	13.20	1.01	1.00
query29	12.62	3.35	3.27
query30	0.26	0.07	0.05
query31	2.88	0.37	0.40
query32	3.30	0.47	0.47
query33	2.89	2.92	2.95
query34	17.12	4.38	4.41
query35	4.49	4.48	4.52
query36	0.66	0.49	0.48
query37	0.18	0.16	0.16
query38	0.15	0.14	0.13
query39	0.04	0.03	0.04
query40	0.19	0.18	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.71 s
Total hot run time: 30.16 s

AshinGau · 2024-06-19T02:44:12Z

run buildall

github-actions · 2024-06-19T02:49:54Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-06-19T03:40:05Z

TPC-H: Total hot run time: 39985 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 37ccf06a7dab99dc4dfb332ed5c581bc3b3d26ca, data reload: false

------ Round 1 ----------------------------------
q1	18007	4847	4439	4439
q2	2697	196	200	196
q3	10987	1121	1144	1121
q4	10357	771	846	771
q5	7480	2731	2656	2656
q6	225	143	139	139
q7	971	625	645	625
q8	9285	2096	2114	2096
q9	8985	6475	6474	6474
q10	8912	3713	3679	3679
q11	465	235	236	235
q12	429	228	229	228
q13	17806	2973	2942	2942
q14	269	218	228	218
q15	519	476	479	476
q16	522	386	372	372
q17	971	713	699	699
q18	8068	7510	7436	7436
q19	8692	1421	1498	1421
q20	653	309	331	309
q21	4907	3115	3213	3115
q22	396	339	338	338
Total cold run time: 121603 ms
Total hot run time: 39985 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4423	4214	4274	4214
q2	366	271	266	266
q3	2978	2736	2695	2695
q4	1866	1563	1612	1563
q5	5249	5281	5282	5281
q6	214	127	129	127
q7	2091	1753	1699	1699
q8	3187	3356	3314	3314
q9	8288	8227	8266	8227
q10	3861	3664	3647	3647
q11	586	493	469	469
q12	769	585	590	585
q13	16348	2976	2980	2976
q14	288	267	248	248
q15	525	481	474	474
q16	461	406	424	406
q17	1760	1471	1487	1471
q18	7704	7398	7303	7303
q19	1707	1499	1579	1499
q20	2025	1777	1752	1752
q21	4904	4758	4739	4739
q22	593	551	521	521
Total cold run time: 70193 ms
Total hot run time: 53476 ms

doris-robot · 2024-06-19T03:51:10Z

TPC-DS: Total hot run time: 172755 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 37ccf06a7dab99dc4dfb332ed5c581bc3b3d26ca, data reload: false

query1	939	384	377	377
query2	6475	2375	2304	2304
query3	6648	205	206	205
query4	19280	17218	17233	17218
query5	4196	479	465	465
query6	250	172	162	162
query7	4585	293	316	293
query8	337	285	295	285
query9	8587	2391	2385	2385
query10	601	290	282	282
query11	10631	9955	9965	9955
query12	140	89	81	81
query13	1642	359	362	359
query14	9439	7712	7552	7552
query15	226	188	185	185
query16	7837	289	262	262
query17	1877	555	521	521
query18	1964	267	270	267
query19	195	156	151	151
query20	94	81	81	81
query21	211	123	123	123
query22	4450	4109	4021	4021
query23	33676	32931	33062	32931
query24	11824	2762	2847	2762
query25	692	382	379	379
query26	1802	160	157	157
query27	2987	329	316	316
query28	7574	2048	2028	2028
query29	1155	653	603	603
query30	282	146	150	146
query31	954	730	757	730
query32	94	55	58	55
query33	773	284	279	279
query34	948	471	457	457
query35	741	606	605	605
query36	1125	929	948	929
query37	190	72	70	70
query38	2854	2730	2720	2720
query39	858	790	783	783
query40	282	125	132	125
query41	55	53	51	51
query42	122	100	102	100
query43	556	527	538	527
query44	1220	715	728	715
query45	194	168	165	165
query46	1066	728	724	724
query47	1849	1758	1766	1758
query48	368	315	286	286
query49	1167	410	413	410
query50	768	392	385	385
query51	6754	6722	6677	6677
query52	102	95	90	90
query53	360	294	297	294
query54	980	435	441	435
query55	74	73	74	73
query56	276	268	259	259
query57	1150	1069	1060	1060
query58	239	235	276	235
query59	3240	3021	3086	3021
query60	295	270	269	269
query61	120	93	90	90
query62	660	440	440	440
query63	322	293	287	287
query64	9853	2227	1716	1716
query65	3393	3111	3135	3111
query66	1380	334	352	334
query67	15165	14895	15128	14895
query68	5381	544	556	544
query69	576	425	376	376
query70	1195	1148	1135	1135
query71	435	264	268	264
query72	7625	5590	5692	5590
query73	773	331	331	331
query74	5808	5442	5468	5442
query75	3758	2651	2620	2620
query76	3220	910	1023	910
query77	671	316	306	306
query78	10329	9652	9748	9652
query79	3009	524	507	507
query80	2095	457	470	457
query81	574	222	215	215
query82	1465	105	100	100
query83	356	174	170	170
query84	270	83	86	83
query85	1567	276	268	268
query86	484	333	321	321
query87	3293	3051	3131	3051
query88	4328	2344	2358	2344
query89	468	388	393	388
query90	1807	206	196	196
query91	130	95	102	95
query92	61	49	49	49
query93	3703	525	514	514
query94	1272	193	197	193
query95	398	316	318	316
query96	594	264	262	262
query97	3240	3043	3015	3015
query98	209	195	191	191
query99	1302	834	860	834
Total cold run time: 281539 ms
Total hot run time: 172755 ms

doris-robot · 2024-06-19T03:56:40Z

ClickBench: Total hot run time: 31.01 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 37ccf06a7dab99dc4dfb332ed5c581bc3b3d26ca, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.68	0.08	0.08
query5	0.50	0.49	0.48
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.55	0.50	0.48
query10	0.56	0.55	0.54
query11	0.16	0.10	0.11
query12	0.14	0.12	0.12
query13	0.59	0.59	0.59
query14	0.75	0.78	0.78
query15	0.82	0.81	0.81
query16	0.35	0.36	0.35
query17	0.95	1.00	0.96
query18	0.20	0.28	0.24
query19	1.80	1.73	1.80
query20	0.01	0.01	0.00
query21	15.43	0.65	0.64
query22	4.27	6.78	2.34
query23	18.28	1.40	1.28
query24	2.09	0.24	0.22
query25	0.16	0.09	0.08
query26	0.27	0.17	0.17
query27	0.08	0.08	0.08
query28	13.26	1.01	0.98
query29	12.56	3.36	3.33
query30	0.25	0.06	0.06
query31	2.88	0.39	0.38
query32	3.26	0.48	0.48
query33	2.90	2.87	2.89
query34	17.02	4.46	4.46
query35	4.54	4.48	4.43
query36	0.66	0.49	0.46
query37	0.18	0.15	0.15
query38	0.15	0.14	0.14
query39	0.05	0.04	0.04
query40	0.17	0.16	0.14
query41	0.10	0.04	0.04
query42	0.07	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.27 s
Total hot run time: 31.01 s

doris-robot · 2024-06-19T04:06:38Z

TeamCity be ut coverage result:
Function Coverage: 36.46% (9003/24695)
Line Coverage: 28.00% (73759/263403)
Region Coverage: 27.48% (38315/139419)
Branch Coverage: 24.17% (19526/80774)
Coverage Report: http://coverage.selectdb-in.cc/coverage/37ccf06a7dab99dc4dfb332ed5c581bc3b3d26ca_37ccf06a7dab99dc4dfb332ed5c581bc3b3d26ca/report/index.html

morningman

LGTM

github-actions · 2024-06-19T08:01:43Z

PR approved by at least one committer and no changes requested.

bp #36045, and turn on batch split, which is turn off in #36109 Generate and get split batch concurrently. `SplitSource.getNextBatch` remove the synchronization, and make each get their splits concurrently, and `SplitAssignment` generates splits asynchronously.

## Proposed changes Generate and get split batch concurrently. `SplitSource.getNextBatch` remove the synchronization, and make each get their splits concurrently, and `SplitAssignment` generates splits asynchronously.

morningman mentioned this pull request Jun 7, 2024

[opt](split) generate and get split batch concurrently #36044

Merged

AshinGau force-pushed the opt_split_m branch from 3b23c53 to 4447726 Compare June 7, 2024 08:39

AshinGau force-pushed the opt_split_m branch from 4447726 to 5b2a01e Compare June 17, 2024 02:34

AshinGau force-pushed the opt_split_m branch from 5b2a01e to dc6b42a Compare June 17, 2024 02:43

morningman previously approved these changes Jun 17, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 17, 2024

github-actions bot added the reviewed label Jun 17, 2024

zddr approved these changes Jun 17, 2024

View reviewed changes

AshinGau dismissed morningman’s stale review via 0df51d5 June 18, 2024 02:48

AshinGau force-pushed the opt_split_m branch from dc6b42a to 0df51d5 Compare June 18, 2024 02:48

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jun 18, 2024

AshinGau force-pushed the opt_split_m branch from 0df51d5 to 5e1e3c6 Compare June 18, 2024 02:51

[opt](split) generate and get split batch concurrently

37ccf06

AshinGau force-pushed the opt_split_m branch from 03123e6 to 37ccf06 Compare June 19, 2024 02:43

morningman approved these changes Jun 19, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 19, 2024

AshinGau merged commit 0400833 into apache:master Jun 19, 2024

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

[opt](split) generate and get split batch concurrently #36045

[opt](split) generate and get split batch concurrently #36045

Uh oh!

Conversation

AshinGau commented Jun 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Improvements

Uh oh!

doris-robot commented Jun 7, 2024

Uh oh!

github-actions bot commented Jun 7, 2024

Uh oh!

AshinGau commented Jun 7, 2024

Uh oh!

github-actions bot commented Jun 7, 2024

Uh oh!

doris-robot commented Jun 7, 2024

Uh oh!

doris-robot commented Jun 7, 2024

Uh oh!

doris-robot commented Jun 7, 2024

Uh oh!

doris-robot commented Jun 7, 2024

Uh oh!

github-actions bot commented Jun 17, 2024

Uh oh!

github-actions bot commented Jun 17, 2024

Uh oh!

AshinGau commented Jun 17, 2024

Uh oh!

doris-robot commented Jun 17, 2024

Uh oh!

doris-robot commented Jun 17, 2024

Uh oh!

doris-robot commented Jun 17, 2024

Uh oh!

doris-robot commented Jun 17, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 17, 2024

Uh oh!

github-actions bot commented Jun 17, 2024

Uh oh!

AshinGau commented Jun 18, 2024

Uh oh!

github-actions bot commented Jun 18, 2024

Uh oh!

github-actions bot commented Jun 18, 2024

Uh oh!

doris-robot commented Jun 18, 2024

Uh oh!

morningman commented Jun 18, 2024

Uh oh!

doris-robot commented Jun 18, 2024

Uh oh!

AshinGau commented Jun 19, 2024

Uh oh!

github-actions bot commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

AshinGau commented Jun 19, 2024

Uh oh!

github-actions bot commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

AshinGau commented Jun 7, 2024 •

edited

Loading