Skip to content

[Refactor](shuffle) Fix wrong deal of batching block in TabletSinkHashPartitioner #48720

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
zclllyybb wants to merge 7 commits into
base: master
Choose a base branch
from
Open

[Refactor](shuffle) Fix wrong deal of batching block in TabletSinkHashPartitioner #48720

zclllyybb wants to merge 7 commits into from
+2,022 −642

Conversation

@zclllyybb
Copy link
Contributor

@zclllyybb zclllyybb commented Mar 5, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: bug introduced by #44102, expanded by #45256, partially fixed by #47911

Problem Summary:

also fixed Auto Partition + Auto Increment-Column may core dump in loading

Release note

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 5, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zclllyybb
Copy link
Contributor Author

zclllyybb commented Mar 5, 2025

run buildall

@zclllyybb zclllyybb force-pushed the fix_resend_ap branch 4 times, most recently from 05c9422 to 5a86da5 Compare March 5, 2025 21:07
@zclllyybb
Copy link
Contributor Author

zclllyybb commented Mar 5, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 5, 2025

TPC-H: Total hot run time: 32869 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5a86da5e6eadb4fbac8772fdb5bfdaaf147f6000, data reload: false

------ Round 1 ----------------------------------
q1	17605	5246	5148	5148
q2	2059	301	177	177
q3	10419	1258	773	773
q4	10253	1026	550	550
q5	7997	2355	2364	2355
q6	188	169	140	140
q7	923	753	602	602
q8	9299	1269	1100	1100
q9	4984	4943	4768	4768
q10	6823	2316	1909	1909
q11	483	285	257	257
q12	343	351	225	225
q13	17784	3703	3114	3114
q14	225	225	208	208
q15	534	493	489	489
q16	618	630	603	603
q17	608	848	386	386
q18	6764	6441	6354	6354
q19	1884	952	559	559
q20	310	329	188	188
q21	2857	2139	1973	1973
q22	1092	1041	991	991
Total cold run time: 104052 ms
Total hot run time: 32869 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5290	5199	5200	5199
q2	241	327	240	240
q3	2188	2699	2358	2358
q4	1410	1847	1344	1344
q5	4240	4116	4178	4116
q6	207	161	123	123
q7	1942	1936	1809	1809
q8	2627	2700	2521	2521
q9	7256	7261	7348	7261
q10	3067	3215	2804	2804
q11	603	524	489	489
q12	655	722	591	591
q13	3528	3924	3361	3361
q14	289	315	279	279
q15	508	491	463	463
q16	641	697	666	666
q17	1158	1580	1347	1347
q18	7755	7635	7552	7552
q19	904	888	987	888
q20	1975	2045	1879	1879
q21	5496	4932	4828	4828
q22	1080	1120	1020	1020
Total cold run time: 53060 ms
Total hot run time: 51138 ms

@doris-robot
Copy link

doris-robot commented Mar 5, 2025

TPC-DS: Total hot run time: 192644 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5a86da5e6eadb4fbac8772fdb5bfdaaf147f6000, data reload: false

query1	1411	1059	1026	1026
query2	6223	2000	1978	1978
query3	10986	4727	4604	4604
query4	53722	24474	23004	23004
query5	5202	525	511	511
query6	388	213	208	208
query7	5127	511	291	291
query8	346	249	240	240
query9	6774	2579	2573	2573
query10	430	339	259	259
query11	15372	15091	14849	14849
query12	170	112	107	107
query13	1174	513	415	415
query14	10571	6846	6515	6515
query15	217	198	179	179
query16	6976	617	474	474
query17	1096	718	570	570
query18	1537	410	335	335
query19	199	200	167	167
query20	130	128	169	128
query21	212	128	113	113
query22	4323	4385	4265	4265
query23	34019	33905	33596	33596
query24	6040	2410	2431	2410
query25	467	487	400	400
query26	764	279	155	155
query27	2018	510	337	337
query28	2815	2511	2453	2453
query29	586	572	430	430
query30	274	220	210	210
query31	893	843	811	811
query32	69	63	60	60
query33	457	359	303	303
query34	796	859	517	517
query35	815	851	782	782
query36	968	1007	905	905
query37	122	103	75	75
query38	4278	4405	4304	4304
query39	1491	1443	1416	1416
query40	215	117	103	103
query41	55	62	62	62
query42	128	115	117	115
query43	535	540	513	513
query44	1395	830	812	812
query45	193	177	172	172
query46	854	1028	652	652
query47	1820	1829	1812	1812
query48	387	411	316	316
query49	720	510	453	453
query50	708	740	444	444
query51	4351	4289	4282	4282
query52	100	106	97	97
query53	238	250	183	183
query54	496	518	451	451
query55	81	84	78	78
query56	303	264	250	250
query57	1173	1194	1141	1141
query58	248	243	247	243
query59	2696	2937	2842	2842
query60	296	296	258	258
query61	129	121	118	118
query62	749	736	698	698
query63	237	197	197	197
query64	2105	1042	719	719
query65	4537	4512	4333	4333
query66	758	391	339	339
query67	15811	15499	15253	15253
query68	7140	888	516	516
query69	540	297	267	267
query70	1235	1134	1137	1134
query71	504	289	277	277
query72	5797	3782	3816	3782
query73	1223	761	341	341
query74	9251	9033	8839	8839
query75	3661	3144	2721	2721
query76	4253	1183	774	774
query77	620	367	273	273
query78	10126	10086	9329	9329
query79	3770	826	564	564
query80	713	522	444	444
query81	495	276	227	227
query82	707	125	93	93
query83	278	166	149	149
query84	288	90	76	76
query85	792	370	310	310
query86	408	292	296	292
query87	4545	4444	4343	4343
query88	3405	2205	2192	2192
query89	430	330	288	288
query90	1936	195	192	192
query91	134	134	109	109
query92	77	61	54	54
query93	2657	1020	569	569
query94	651	414	293	293
query95	353	269	266	266
query96	489	563	266	266
query97	3338	3436	3287	3287
query98	248	200	195	195
query99	1450	1407	1271	1271
Total cold run time: 302290 ms
Total hot run time: 192644 ms

@doris-robot
Copy link

doris-robot commented Mar 5, 2025

ClickBench: Total hot run time: 31.03 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5a86da5e6eadb4fbac8772fdb5bfdaaf147f6000, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.03
query3	0.24	0.06	0.07
query4	1.62	0.11	0.10
query5	0.56	0.54	0.55
query6	1.18	0.72	0.72
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.52	0.53
query10	0.58	0.57	0.59
query11	0.16	0.10	0.11
query12	0.15	0.12	0.12
query13	0.62	0.61	0.59
query14	2.72	2.71	2.69
query15	0.93	0.87	0.86
query16	0.40	0.38	0.37
query17	1.03	1.03	1.01
query18	0.21	0.19	0.19
query19	1.89	1.79	1.97
query20	0.01	0.01	0.02
query21	15.35	0.92	0.57
query22	0.76	1.10	0.59
query23	15.10	1.36	0.62
query24	7.25	1.12	1.61
query25	0.52	0.30	0.14
query26	0.61	0.16	0.14
query27	0.05	0.05	0.05
query28	10.02	0.79	0.42
query29	12.58	3.91	3.32
query30	0.26	0.09	0.07
query31	3.09	0.58	0.39
query32	3.23	0.55	0.46
query33	2.98	2.97	2.96
query34	15.74	5.10	4.53
query35	4.50	4.54	4.52
query36	0.68	0.50	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.14	0.12
query41	0.09	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.29 s
Total hot run time: 31.03 s

@doris-robot
Copy link

doris-robot commented Mar 5, 2025

BE UT Coverage Report

Increment line coverage 3.00% (3/100) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.93% (12278/26732)
Line Coverage 35.39% (103659/292945)
Region Coverage 34.55% (53097/153701)
Branch Coverage 30.23% (26882/88914)

yiguolei
yiguolei reviewed Mar 6, 2025
View reviewed changes
Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These files should add BE UT

Gabriel39
Gabriel39 requested changes Mar 6, 2025
View reviewed changes
@zclllyybb
Copy link
Contributor Author

zclllyybb commented Mar 6, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 6, 2025

TPC-H: Total hot run time: 32475 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cd24ddd638a82398053c2992892186a232d92554, data reload: false

------ Round 1 ----------------------------------
q1	17587	5135	5028	5028
q2	2041	292	183	183
q3	10438	1288	769	769
q4	10262	1020	525	525
q5	7955	2429	2342	2342
q6	197	166	131	131
q7	921	747	594	594
q8	9298	1275	1087	1087
q9	4997	4639	4769	4639
q10	6818	2338	1899	1899
q11	461	270	263	263
q12	349	354	219	219
q13	17766	3697	3051	3051
q14	236	217	203	203
q15	542	506	498	498
q16	654	615	604	604
q17	562	868	350	350
q18	6759	6561	6373	6373
q19	1915	941	543	543
q20	308	321	190	190
q21	2786	2177	1953	1953
q22	1097	1043	1031	1031
Total cold run time: 103949 ms
Total hot run time: 32475 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5179	5118	5097	5097
q2	232	332	233	233
q3	2140	2677	2247	2247
q4	1440	1904	1391	1391
q5	4270	4082	4185	4082
q6	208	165	122	122
q7	1884	1887	1737	1737
q8	2650	2704	2583	2583
q9	7283	7220	7156	7156
q10	3033	3254	2774	2774
q11	581	513	491	491
q12	728	724	568	568
q13	3457	3968	3269	3269
q14	274	302	282	282
q15	520	492	478	478
q16	648	685	623	623
q17	1146	1629	1312	1312
q18	7875	7726	7574	7574
q19	825	781	884	781
q20	1965	2038	1900	1900
q21	5494	4922	4675	4675
q22	1112	1076	1083	1076
Total cold run time: 52944 ms
Total hot run time: 50451 ms

@doris-robot
Copy link

doris-robot commented Mar 6, 2025

TPC-DS: Total hot run time: 192865 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cd24ddd638a82398053c2992892186a232d92554, data reload: false

query1	1359	996	1025	996
query2	6091	1966	1954	1954
query3	11162	4613	4608	4608
query4	56298	25563	23393	23393
query5	4950	548	493	493
query6	339	207	200	200
query7	4891	502	289	289
query8	335	243	236	236
query9	5741	2602	2602	2602
query10	431	306	272	272
query11	15308	15106	14977	14977
query12	149	107	110	107
query13	1079	531	396	396
query14	10571	6772	6305	6305
query15	193	198	175	175
query16	7062	671	492	492
query17	1060	698	574	574
query18	1521	406	318	318
query19	204	200	166	166
query20	128	131	124	124
query21	214	128	106	106
query22	4584	4516	4274	4274
query23	34022	33512	33450	33450
query24	5813	2391	2420	2391
query25	461	465	405	405
query26	666	280	160	160
query27	1665	497	331	331
query28	2787	2495	2478	2478
query29	591	552	442	442
query30	273	219	198	198
query31	867	875	806	806
query32	68	65	63	63
query33	436	357	343	343
query34	762	872	514	514
query35	831	852	769	769
query36	931	1009	921	921
query37	118	97	77	77
query38	4144	4195	4310	4195
query39	1460	1463	1430	1430
query40	221	121	110	110
query41	63	56	62	56
query42	122	108	104	104
query43	502	538	497	497
query44	1331	837	826	826
query45	193	179	167	167
query46	873	1044	669	669
query47	1860	1919	1814	1814
query48	413	423	316	316
query49	697	532	429	429
query50	712	756	438	438
query51	4286	4286	4288	4286
query52	115	102	90	90
query53	234	278	193	193
query54	501	514	421	421
query55	125	83	81	81
query56	284	275	236	236
query57	1172	1184	1132	1132
query58	248	240	228	228
query59	2769	2887	2871	2871
query60	287	281	261	261
query61	119	123	124	123
query62	740	743	688	688
query63	250	189	194	189
query64	1456	1057	700	700
query65	4526	4467	4416	4416
query66	758	401	347	347
query67	16005	15777	15367	15367
query68	6940	877	512	512
query69	538	305	263	263
query70	1182	1132	1086	1086
query71	506	285	272	272
query72	5814	3767	3780	3767
query73	1363	760	346	346
query74	9176	8902	8827	8827
query75	3823	3161	2702	2702
query76	4170	1204	760	760
query77	641	373	282	282
query78	10122	9995	9293	9293
query79	4460	823	572	572
query80	728	576	458	458
query81	498	260	225	225
query82	692	130	93	93
query83	293	172	147	147
query84	286	92	75	75
query85	788	349	321	321
query86	392	309	309	309
query87	4536	4452	4419	4419
query88	3540	2222	2250	2222
query89	451	308	282	282
query90	1821	213	214	213
query91	139	135	109	109
query92	80	57	58	57
query93	3094	1031	567	567
query94	661	425	302	302
query95	400	270	266	266
query96	487	569	288	288
query97	3345	3425	3284	3284
query98	231	204	204	204
query99	1432	1403	1257	1257
Total cold run time: 303056 ms
Total hot run time: 192865 ms

@doris-robot
Copy link

doris-robot commented Mar 6, 2025

ClickBench: Total hot run time: 30.96 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cd24ddd638a82398053c2992892186a232d92554, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.07
query4	1.62	0.10	0.10
query5	0.57	0.56	0.54
query6	1.18	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.59	0.53	0.53
query10	0.58	0.60	0.58
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.62	0.61	0.60
query14	2.74	2.71	2.85
query15	0.92	0.85	0.86
query16	0.38	0.39	0.40
query17	1.04	1.06	1.01
query18	0.22	0.19	0.20
query19	1.90	1.88	2.00
query20	0.02	0.01	0.02
query21	15.35	0.90	0.53
query22	0.76	1.19	0.78
query23	14.76	1.35	0.63
query24	6.62	1.45	0.73
query25	0.54	0.28	0.07
query26	0.49	0.17	0.15
query27	0.06	0.06	0.05
query28	9.24	0.89	0.44
query29	12.53	4.03	3.36
query30	0.25	0.09	0.06
query31	2.81	0.59	0.39
query32	3.23	0.56	0.46
query33	3.02	3.00	3.06
query34	15.82	5.17	4.50
query35	4.59	4.58	4.52
query36	0.66	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.03
query40	0.16	0.13	0.14
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.29 s
Total hot run time: 30.96 s

@zclllyybb zclllyybb force-pushed the fix_resend_ap branch 2 times, most recently from a1aa63a to e74e9c5 Compare May 14, 2025 07:36
@zclllyybb
Copy link
Contributor Author

zclllyybb commented May 14, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 14, 2025

TPC-H: Total hot run time: 33882 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e74e9c5bd79cf89dff7711040c93f57cf3789812, data reload: false

------ Round 1 ----------------------------------
q1	26181	5022	4947	4947
q2	2062	270	182	182
q3	10484	1241	724	724
q4	10227	1016	562	562
q5	7753	2410	2345	2345
q6	189	165	133	133
q7	925	782	618	618
q8	9322	1275	1047	1047
q9	6791	5073	5079	5073
q10	6848	2327	1886	1886
q11	481	283	264	264
q12	342	345	213	213
q13	17777	3691	3097	3097
q14	238	230	219	219
q15	537	488	495	488
q16	416	426	380	380
q17	600	860	368	368
q18	7632	7322	7071	7071
q19	1476	947	567	567
q20	335	354	222	222
q21	4037	3412	2488	2488
q22	1070	1023	988	988
Total cold run time: 115723 ms
Total hot run time: 33882 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5123	5081	5043	5043
q2	233	350	234	234
q3	2166	2650	2314	2314
q4	1435	1839	1386	1386
q5	4567	4443	4425	4425
q6	212	167	128	128
q7	1965	1887	1750	1750
q8	2592	2523	2468	2468
q9	7175	7030	7034	7030
q10	3024	3156	2756	2756
q11	577	523	514	514
q12	688	755	590	590
q13	3535	3865	3318	3318
q14	283	292	268	268
q15	516	481	462	462
q16	452	483	430	430
q17	1138	1575	1375	1375
q18	7797	7462	7337	7337
q19	791	773	853	773
q20	1989	2084	1965	1965
q21	5126	4708	4576	4576
q22	1074	1048	1011	1011
Total cold run time: 52458 ms
Total hot run time: 50153 ms

@doris-robot
Copy link

doris-robot commented May 14, 2025

TPC-DS: Total hot run time: 187015 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e74e9c5bd79cf89dff7711040c93f57cf3789812, data reload: false

query1	1017	473	477	473
query2	6560	1908	1848	1848
query3	6769	221	221	221
query4	26342	24121	23530	23530
query5	4655	618	458	458
query6	288	196	222	196
query7	4625	503	289	289
query8	283	240	224	224
query9	8636	2655	2654	2654
query10	513	330	273	273
query11	15357	15076	14845	14845
query12	155	107	105	105
query13	1653	530	410	410
query14	8728	6044	6154	6044
query15	216	188	171	171
query16	7134	644	479	479
query17	1082	729	584	584
query18	1976	403	308	308
query19	197	188	176	176
query20	119	119	120	119
query21	214	122	108	108
query22	4114	4087	4084	4084
query23	34446	33170	32987	32987
query24	8109	2467	2385	2385
query25	546	443	395	395
query26	1231	267	156	156
query27	2744	493	336	336
query28	4345	2133	2103	2103
query29	759	553	431	431
query30	290	214	193	193
query31	978	861	759	759
query32	73	65	63	63
query33	556	375	303	303
query34	799	861	506	506
query35	778	799	732	732
query36	924	982	885	885
query37	111	104	75	75
query38	4164	4227	4231	4227
query39	1468	1391	1406	1391
query40	214	118	106	106
query41	57	56	53	53
query42	123	110	106	106
query43	505	503	460	460
query44	1281	807	794	794
query45	175	174	163	163
query46	844	1022	624	624
query47	1777	1814	1735	1735
query48	380	409	314	314
query49	788	523	417	417
query50	644	679	394	394
query51	4152	4107	4146	4107
query52	109	109	96	96
query53	219	256	194	194
query54	615	567	504	504
query55	81	79	84	79
query56	293	300	296	296
query57	1160	1148	1122	1122
query58	260	266	249	249
query59	2623	2644	2643	2643
query60	315	305	315	305
query61	120	124	118	118
query62	788	741	664	664
query63	226	187	186	186
query64	4383	1010	653	653
query65	4355	4232	4318	4232
query66	1170	448	333	333
query67	15894	15424	15815	15424
query68	8058	889	512	512
query69	496	292	325	292
query70	1277	1143	1106	1106
query71	470	320	294	294
query72	5778	4773	4812	4773
query73	717	617	342	342
query74	9226	9168	9007	9007
query75	3783	3223	2737	2737
query76	3622	1192	743	743
query77	791	380	287	287
query78	10157	9982	9369	9369
query79	2725	827	572	572
query80	621	536	448	448
query81	481	267	224	224
query82	550	131	98	98
query83	291	248	241	241
query84	298	109	83	83
query85	805	355	316	316
query86	387	298	305	298
query87	4541	4456	4433	4433
query88	3593	2310	2241	2241
query89	378	329	277	277
query90	1814	201	261	201
query91	135	159	113	113
query92	76	66	56	56
query93	1820	947	572	572
query94	669	418	304	304
query95	377	292	277	277
query96	503	553	277	277
query97	3182	3217	3119	3119
query98	233	209	206	206
query99	1438	1398	1300	1300
Total cold run time: 275797 ms
Total hot run time: 187015 ms

@doris-robot
Copy link

doris-robot commented May 14, 2025

ClickBench: Total hot run time: 29.84 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e74e9c5bd79cf89dff7711040c93f57cf3789812, data reload: false

query1	0.04	0.03	0.03
query2	0.12	0.10	0.10
query3	0.26	0.20	0.19
query4	1.59	0.20	0.19
query5	0.60	0.57	0.60
query6	1.19	0.72	0.72
query7	0.02	0.02	0.02
query8	0.05	0.03	0.03
query9	0.57	0.54	0.52
query10	0.58	0.58	0.57
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.61	0.60
query14	0.79	0.82	0.80
query15	0.87	0.85	0.84
query16	0.38	0.38	0.38
query17	1.07	1.06	1.02
query18	0.23	0.21	0.21
query19	1.88	1.84	1.90
query20	0.01	0.01	0.01
query21	15.42	0.91	0.57
query22	0.75	1.16	0.61
query23	15.06	1.37	0.61
query24	6.58	1.37	1.24
query25	0.46	0.16	0.26
query26	0.66	0.16	0.13
query27	0.05	0.05	0.05
query28	9.40	0.90	0.44
query29	12.54	3.98	3.41
query30	0.25	0.11	0.06
query31	2.82	0.60	0.39
query32	3.23	0.55	0.46
query33	3.11	3.06	3.14
query34	15.94	5.16	4.52
query35	4.53	4.53	4.50
query36	0.66	0.49	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.12	0.13
query41	0.09	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.09 s
Total hot run time: 29.84 s

@zclllyybb
Copy link
Contributor Author

zclllyybb commented May 14, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 14, 2025

TPC-H: Total hot run time: 33901 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0c6f154d765942727ba133703243016ae76a32ce, data reload: false

------ Round 1 ----------------------------------
q1	26061	5086	5062	5062
q2	2074	277	192	192
q3	10394	1245	694	694
q4	10223	1016	535	535
q5	7541	2405	2361	2361
q6	191	170	138	138
q7	904	731	620	620
q8	9337	1298	1103	1103
q9	6800	5199	5055	5055
q10	6877	2331	1897	1897
q11	490	289	273	273
q12	351	349	238	238
q13	17793	3702	3164	3164
q14	231	226	209	209
q15	528	493	512	493
q16	427	432	369	369
q17	604	868	361	361
q18	7536	7184	7071	7071
q19	1621	966	558	558
q20	345	346	212	212
q21	4191	2616	2327	2327
q22	1054	989	969	969
Total cold run time: 115573 ms
Total hot run time: 33901 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5264	5132	5133	5132
q2	237	316	224	224
q3	2179	2674	2310	2310
q4	1363	1795	1339	1339
q5	4429	4375	4431	4375
q6	212	170	127	127
q7	2001	1962	1778	1778
q8	2619	2678	2541	2541
q9	7130	7139	6962	6962
q10	3073	3174	2786	2786
q11	598	532	495	495
q12	703	791	641	641
q13	3478	3918	3288	3288
q14	299	292	273	273
q15	558	531	505	505
q16	440	472	455	455
q17	1173	1582	1414	1414
q18	7723	7535	7430	7430
q19	862	875	892	875
q20	1968	2051	1854	1854
q21	4859	4478	4512	4478
q22	1102	1069	1014	1014
Total cold run time: 52270 ms
Total hot run time: 50296 ms

@doris-robot
Copy link

doris-robot commented May 14, 2025

TPC-DS: Total hot run time: 193904 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0c6f154d765942727ba133703243016ae76a32ce, data reload: false

query1	1428	1099	1054	1054
query2	6224	1796	1754	1754
query3	11003	4532	4538	4532
query4	54859	25840	23573	23573
query5	5003	509	457	457
query6	345	230	193	193
query7	4867	520	302	302
query8	311	248	244	244
query9	5335	2681	2676	2676
query10	451	340	275	275
query11	15047	15091	14769	14769
query12	168	113	103	103
query13	1033	543	408	408
query14	10233	6400	6434	6400
query15	206	213	185	185
query16	7118	656	522	522
query17	1101	760	617	617
query18	1623	413	305	305
query19	201	199	162	162
query20	122	116	123	116
query21	208	123	109	109
query22	4404	4600	4303	4303
query23	34359	33689	33805	33689
query24	6558	2469	2471	2469
query25	463	482	414	414
query26	717	277	148	148
query27	2303	517	339	339
query28	3075	2199	2168	2168
query29	584	587	473	473
query30	278	226	192	192
query31	805	861	777	777
query32	79	69	60	60
query33	459	358	306	306
query34	792	860	547	547
query35	794	875	738	738
query36	951	995	890	890
query37	115	101	73	73
query38	4253	4261	4196	4196
query39	1480	1483	1428	1428
query40	208	125	108	108
query41	57	55	52	52
query42	126	112	107	107
query43	504	520	466	466
query44	1349	860	829	829
query45	179	177	167	167
query46	856	1052	640	640
query47	1871	1878	1812	1812
query48	415	434	334	334
query49	696	544	449	449
query50	673	707	403	403
query51	4299	4272	4264	4264
query52	114	114	109	109
query53	231	271	197	197
query54	598	615	538	538
query55	86	80	88	80
query56	326	315	275	275
query57	1213	1217	1124	1124
query58	253	258	256	256
query59	2640	2766	2728	2728
query60	342	356	325	325
query61	123	121	121	121
query62	737	721	701	701
query63	240	192	197	192
query64	1942	1042	737	737
query65	4305	4276	4221	4221
query66	721	406	306	306
query67	15975	15784	15495	15495
query68	7179	913	520	520
query69	541	311	272	272
query70	1197	1139	1133	1133
query71	506	334	296	296
query72	5862	4676	4709	4676
query73	1512	569	348	348
query74	8982	9034	8809	8809
query75	3880	3244	2762	2762
query76	4132	1190	747	747
query77	641	373	291	291
query78	10198	10122	9352	9352
query79	2395	824	588	588
query80	641	505	442	442
query81	487	249	221	221
query82	269	135	94	94
query83	249	245	228	228
query84	310	104	89	89
query85	814	364	308	308
query86	347	297	308	297
query87	4473	4492	4331	4331
query88	3739	2319	2322	2319
query89	408	320	291	291
query90	1763	215	221	215
query91	145	146	112	112
query92	75	64	59	59
query93	1939	954	585	585
query94	614	414	306	306
query95	371	293	286	286
query96	520	568	293	293
query97	3166	3176	3167	3167
query98	229	215	210	210
query99	1350	1408	1255	1255
Total cold run time: 299368 ms
Total hot run time: 193904 ms

@doris-robot
Copy link

doris-robot commented May 14, 2025

ClickBench: Total hot run time: 29.78 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0c6f154d765942727ba133703243016ae76a32ce, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.12
query3	0.25	0.19	0.20
query4	1.59	0.19	0.11
query5	0.57	0.55	0.56
query6	1.21	0.72	0.73
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.58	0.54	0.50
query10	0.56	0.57	0.57
query11	0.16	0.11	0.10
query12	0.15	0.11	0.12
query13	0.62	0.60	0.60
query14	0.79	0.80	0.83
query15	0.88	0.85	0.85
query16	0.38	0.39	0.40
query17	1.02	1.02	1.03
query18	0.22	0.21	0.21
query19	1.91	1.84	1.81
query20	0.02	0.01	0.01
query21	15.42	0.90	0.55
query22	0.76	1.17	0.64
query23	14.98	1.36	0.61
query24	6.79	1.52	1.41
query25	0.49	0.30	0.06
query26	0.60	0.15	0.15
query27	0.05	0.05	0.04
query28	9.16	0.91	0.45
query29	12.57	3.98	3.38
query30	0.25	0.09	0.07
query31	2.81	0.59	0.39
query32	3.23	0.54	0.46
query33	3.02	3.10	3.04
query34	15.57	5.15	4.50
query35	4.51	4.54	4.48
query36	0.68	0.50	0.48
query37	0.08	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.09	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 102.51 s
Total hot run time: 29.78 s

@zclllyybb
Copy link
Contributor Author

zclllyybb commented May 16, 2025

run buildall

@zclllyybb
Copy link
Contributor Author

zclllyybb commented Jan 20, 2026

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 20, 2026

FE UT Coverage Report

Increment line coverage 100% (0/0) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

doris-robot commented Jan 20, 2026

TPC-H: Total hot run time: 31965 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8a0912722d6d6ff55a76f81fb25d6916191ec602, data reload: false

------ Round 1 ----------------------------------
q1	17628	4214	4036	4036
q2	2061	344	245	245
q3	10163	1287	734	734
q4	10216	940	311	311
q5	7544	2131	1867	1867
q6	185	168	136	136
q7	945	782	646	646
q8	9290	1409	1138	1138
q9	5021	4586	4604	4586
q10	6799	1810	1411	1411
q11	530	298	289	289
q12	717	721	614	614
q13	17799	3815	3112	3112
q14	292	313	290	290
q15	590	516	512	512
q16	701	686	620	620
q17	664	780	524	524
q18	6896	6454	7052	6454
q19	1144	1025	656	656
q20	408	370	248	248
q21	3277	2636	2541	2541
q22	1118	1120	995	995
Total cold run time: 103988 ms
Total hot run time: 31965 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4431	4338	4174	4174
q2	358	396	324	324
q3	2309	2808	2325	2325
q4	1423	1846	1482	1482
q5	4423	4503	4372	4372
q6	222	176	134	134
q7	2077	1914	1707	1707
q8	2549	2380	2540	2380
q9	7133	7244	7079	7079
q10	2583	2756	2340	2340
q11	550	462	452	452
q12	728	754	627	627
q13	3596	4139	3072	3072
q14	271	278	249	249
q15	517	477	473	473
q16	610	654	611	611
q17	1098	1337	1352	1337
q18	7364	7285	7325	7285
q19	820	796	808	796
q20	1904	1987	1761	1761
q21	4516	4176	4103	4103
q22	1086	1052	959	959
Total cold run time: 50568 ms
Total hot run time: 48042 ms

@doris-robot
Copy link

doris-robot commented Jan 20, 2026

TPC-DS: Total hot run time: 173427 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8a0912722d6d6ff55a76f81fb25d6916191ec602, data reload: false

query5	4386	619	484	484
query6	323	221	200	200
query7	4212	464	259	259
query8	339	237	237	237
query9	8716	2872	2913	2872
query10	509	369	327	327
query11	15293	15102	15077	15077
query12	190	115	110	110
query13	1250	480	383	383
query14	6288	2990	2758	2758
query14_1	2624	2648	2654	2648
query15	208	192	171	171
query16	974	474	477	474
query17	1099	687	576	576
query18	2531	436	334	334
query19	236	216	204	204
query20	131	116	111	111
query21	214	134	117	117
query22	3855	4117	3914	3914
query23	16011	15630	15399	15399
query23_1	15429	15429	15573	15429
query24	7235	1550	1162	1162
query24_1	1165	1171	1190	1171
query25	532	455	436	436
query26	1225	261	147	147
query27	2772	438	269	269
query28	4598	2178	2145	2145
query29	726	515	408	408
query30	315	246	205	205
query31	759	628	574	574
query32	96	84	72	72
query33	523	369	295	295
query34	911	867	517	517
query35	711	760	665	665
query36	876	911	774	774
query37	134	99	84	84
query38	2728	2782	2600	2600
query39	787	749	736	736
query39_1	739	702	733	702
query40	227	140	119	119
query41	64	66	64	64
query42	102	97	95	95
query43	472	465	399	399
query44	1333	765	745	745
query45	185	188	177	177
query46	825	959	562	562
query47	1434	1539	1341	1341
query48	317	326	231	231
query49	605	439	376	376
query50	617	269	201	201
query51	3771	3785	3735	3735
query52	100	104	98	98
query53	285	318	267	267
query54	285	274	250	250
query55	79	78	75	75
query56	310	298	295	295
query57	1011	1000	967	967
query58	261	252	258	252
query59	2012	2075	1938	1938
query60	336	332	314	314
query61	149	144	145	144
query62	379	357	313	313
query63	290	261	267	261
query64	4866	1255	940	940
query65	3810	3744	3755	3744
query66	1427	424	312	312
query67	15561	15556	15364	15364
query68	2423	1105	770	770
query69	448	363	321	321
query70	1008	946	940	940
query71	323	313	273	273
query72	5295	3166	3448	3166
query73	614	726	325	325
query74	8741	8793	8537	8537
query75	2748	2810	2452	2452
query76	2284	1045	679	679
query77	372	399	312	312
query78	9720	10056	9187	9187
query79	1068	906	590	590
query80	886	593	528	528
query81	507	265	231	231
query82	1323	147	112	112
query83	328	267	240	240
query84	249	112	93	93
query85	858	469	416	416
query86	352	296	271	271
query87	2867	2878	2749	2749
query88	3452	2598	2567	2567
query89	387	339	326	326
query90	1910	172	159	159
query91	167	155	133	133
query92	71	75	70	70
query93	946	891	547	547
query94	552	314	288	288
query95	584	341	385	341
query96	639	500	228	228
query97	2349	2352	2303	2303
query98	215	202	201	201
query99	612	556	551	551
Total cold run time: 246484 ms
Total hot run time: 173427 ms

@doris-robot
Copy link

doris-robot commented Jan 20, 2026

ClickBench: Total hot run time: 26.51 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8a0912722d6d6ff55a76f81fb25d6916191ec602, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.05	0.05
query3	0.25	0.09	0.09
query4	1.61	0.11	0.11
query5	0.27	0.27	0.25
query6	1.14	0.66	0.64
query7	0.03	0.03	0.02
query8	0.05	0.04	0.04
query9	0.58	0.49	0.49
query10	0.55	0.54	0.55
query11	0.14	0.10	0.09
query12	0.14	0.10	0.10
query13	0.60	0.59	0.57
query14	0.94	0.92	0.94
query15	0.80	0.77	0.76
query16	0.39	0.39	0.41
query17	0.98	1.00	1.04
query18	0.23	0.21	0.20
query19	1.86	1.82	1.78
query20	0.02	0.01	0.02
query21	15.46	0.25	0.14
query22	5.35	0.06	0.04
query23	16.25	0.27	0.10
query24	1.54	0.85	0.28
query25	0.11	0.10	0.08
query26	0.14	0.14	0.13
query27	0.06	0.04	0.06
query28	4.13	1.06	0.90
query29	12.58	3.88	3.14
query30	0.27	0.14	0.12
query31	2.82	0.61	0.38
query32	3.23	0.55	0.46
query33	3.03	2.99	3.02
query34	15.99	5.06	4.40
query35	4.53	4.36	4.41
query36	0.65	0.50	0.49
query37	0.10	0.06	0.07
query38	0.08	0.04	0.04
query39	0.05	0.03	0.03
query40	0.16	0.15	0.14
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.05	0.03	0.03
Total cold run time: 97.44 s
Total hot run time: 26.51 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 20, 2026

BE UT Coverage Report

Increment line coverage 53.53% (182/340) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.62% (19135/36363)
Line Coverage 35.92% (177559/494344)
Region Coverage 32.47% (137634/423930)
Branch Coverage 33.36% (59486/178329)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 20, 2026

BE Regression && UT Coverage Report

Increment line coverage 90.86% (308/339) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.11% (26058/35642)
Line Coverage 56.11% (277085/493792)
Region Coverage 53.89% (230889/428436)
Branch Coverage 55.56% (99502/179092)

@BiteTheDDDDt BiteTheDDDDt requested a review from Copilot January 23, 2026 03:18
BiteTheDDDDt
BiteTheDDDDt reviewed Jan 23, 2026
View reviewed changes
Copilot AI reviewed Jan 23, 2026
View reviewed changes
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 34 out of 36 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

yiguolei
yiguolei reviewed Jan 23, 2026
View reviewed changes
Mryange
Mryange reviewed Jan 23, 2026
View reviewed changes
Mryange
Mryange reviewed Jan 23, 2026
View reviewed changes
Mryange
Mryange reviewed Jan 23, 2026
View reviewed changes
Mryange
Mryange reviewed Jan 23, 2026
View reviewed changes
Mryange
Mryange reviewed Jan 23, 2026
View reviewed changes
@zclllyybb
Copy link
Contributor Author

zclllyybb commented Jan 23, 2026

run buildall

@doris-robot
Copy link

doris-robot commented Jan 23, 2026

TPC-DS: Total hot run time: 0 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f03e812894dcf2f3e994fa427b9cb4b5e840936c, data reload: false

query5	
query6	
query7	
query8	
query9	
query10	
query11	
query12	
query13	
query14	
query14_1	
query15	
query16	
query17	
query18	
query19	
query20	
query21	
query22	
query23	
query23_1	
query24	
query24_1	
query25	
query26	
query27	
query28	
query29	
query30	
query31	
query32	
query33	
query34	
query35	
query36	
query37	
query38	
query39	
query39_1	
query40	
query41	
query42	
query43	
query44	
query45	
query46	
query47	
query48	
query49	
query50	
query51	
query52	
query53	
query54	
query55	
query56	
query57	
query58	
query59	
query60	
query61	
query62	
query63	
query64	
query65	
query66	
query67	
query68	
query69	
query70	
query71	
query72	
query73	
query74	
query75	
query76	
query77	
query78	
query79	
query80	
query81	
query82	
query83	
query84	
query85	
query86	
query87	
query88	
query89	
query90	
query91	
query92	
query93	
query94	
query95	
query96	
query97	
query98	
query99	
Total cold run time: 0 ms
Total hot run time: 0 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 23, 2026

BE UT Coverage Report

Increment line coverage 55.10% (200/363) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.69% (19208/36452)
Line Coverage 36.05% (178727/495783)
Region Coverage 32.52% (138263/425177)
Branch Coverage 33.45% (59833/178887)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 24, 2026

BE Regression && UT Coverage Report

Increment line coverage 59.39% (215/362) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.97% (20355/35731)
Line Coverage 39.80% (197095/495257)
Region Coverage 36.32% (156060/429693)
Branch Coverage 36.95% (66393/179660)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 24, 2026

FE Regression Coverage Report

Increment line coverage 100% (0/0) 🎉
Increment coverage report
Complete coverage report

@zclllyybb
Copy link
Contributor Author

zclllyybb commented Jan 24, 2026

run buildall

@doris-robot
Copy link

doris-robot commented Jan 24, 2026

TPC-DS: Total hot run time: 0 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3572829cf4940deff93b17329e47d968376303d4, data reload: false

query5	
query6	
query7	
query8	
query9	
query10	
query11	
query12	
query13	
query14	
query14_1	
query15	
query16	
query17	
query18	
query19	
query20	
query21	
query22	
query23	
query23_1	
query24	
query24_1	
query25	
query26	
query27	
query28	
query29	
query30	
query31	
query32	
query33	
query34	
query35	
query36	
query37	
query38	
query39	
query39_1	
query40	
query41	
query42	
query43	
query44	
query45	
query46	
query47	
query48	
query49	
query50	
query51	
query52	
query53	
query54	
query55	
query56	
query57	
query58	
query59	
query60	
query61	
query62	
query63	
query64	
query65	
query66	
query67	
query68	
query69	
query70	
query71	
query72	
query73	
query74	
query75	
query76	
query77	
query78	
query79	
query80	
query81	
query82	
query83	
query84	
query85	
query86	
query87	
query88	
query89	
query90	
query91	
query92	
query93	
query94	
query95	
query96	
query97	
query98	
query99	
Total cold run time: 0 ms
Total hot run time: 0 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 24, 2026

BE UT Coverage Report

Increment line coverage 54.97% (199/362) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.73% (19221/36452)
Line Coverage 36.07% (178834/495782)
Region Coverage 32.55% (138387/425177)
Branch Coverage 33.47% (59876/178887)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 24, 2026

BE Regression && UT Coverage Report

Increment line coverage 59.28% (214/361) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 57.03% (20377/35731)
Line Coverage 39.85% (197370/495256)
Region Coverage 36.56% (157117/429708)
Branch Coverage 37.07% (66606/179662)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 24, 2026

FE Regression Coverage Report

Increment line coverage 100% (0/0) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BiteTheDDDDt BiteTheDDDDt BiteTheDDDDt left review comments

@yiguolei yiguolei yiguolei left review comments

Copilot code review Copilot Copilot left review comments

@HappenLee HappenLee HappenLee left review comments

@Gabriel39 Gabriel39 Awaiting requested review from Gabriel39

+1 more reviewer

@Mryange Mryange Mryange left review comments

Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

dev/4.0.x reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@zclllyybb @hello-stephen @doris-robot @BiteTheDDDDt @yiguolei @HappenLee @Gabriel39 @Mryange