Skip to content

branch-3.1: [enhancement](memory) Add configurable threshold for small memory task protection #56994

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 9 commits into from
Oct 28, 2025
Merged

branch-3.1: [enhancement](memory) Add configurable threshold for small memory task protection #56994

morrySnow merged 9 commits into from
Oct 28, 2025

Conversation

@wenzhenghu
Copy link
Contributor

@wenzhenghu wenzhenghu commented Oct 15, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #56104
only branch 3.0/3.1 have this issue, so we just put pr direct to branch-3.1

Changes:

  • Add new BE config parameter mem_tracker_limit_small_memory_task_bytes with default
    value of 32MB (33554432 bytes)
  • Replace hardcoded 32MB threshold in free_top_overcommit_query with the configurable
    parameter
  • Add small memory task protection logic in free_top_memory_query function to ensure
    consistent behavior across both memory release mechanisms

This commit introduces a configurable parameter mem_tracker_limit_small_memory_task_bytes
to replace the hardcoded 32MB threshold for protecting small memory tasks from being
cancelled during memory pressure.

This improvement allows administrators to adjust the small memory task threshold based on
their deployment environment and requirements, while maintaining backward compatibility
with the existing 32MB default value.

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@wenzhenghu wenzhenghu requested a review from morrySnow as a code owner October 15, 2025 08:37
@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 15, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@wenzhenghu
Copy link
Contributor Author

wenzhenghu commented Oct 15, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 15, 2025

TPC-H: Total hot run time: 32943 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ff3256c223eecdbfb192c8226fdf896046562d00, data reload: false

------ Round 1 ----------------------------------
q1	17967	5509	5567	5509
q2	2025	408	283	283
q3	12453	1262	777	777
q4	10437	889	455	455
q5	9304	2369	2117	2117
q6	183	162	133	133
q7	910	741	610	610
q8	9323	1450	1169	1169
q9	5258	4960	4919	4919
q10	6782	2271	1849	1849
q11	490	284	264	264
q12	346	351	210	210
q13	17769	3605	3061	3061
q14	222	226	218	218
q15	524	469	460	460
q16	421	433	374	374
q17	592	869	367	367
q18	7156	6380	6415	6380
q19	1221	951	578	578
q20	339	342	203	203
q21	2932	2232	2005	2005
q22	1093	1032	1002	1002
Total cold run time: 107747 ms
Total hot run time: 32943 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5566	5556	5550	5550
q2	245	323	241	241
q3	2265	2645	2327	2327
q4	1389	1876	1380	1380
q5	4414	5099	5010	5010
q6	174	173	133	133
q7	2096	1995	1801	1801
q8	2637	2839	2775	2775
q9	7273	7297	7225	7225
q10	3030	3202	2810	2810
q11	600	511	493	493
q12	679	772	601	601
q13	3424	3815	3134	3134
q14	275	294	283	283
q15	531	480	476	476
q16	428	473	443	443
q17	1265	1751	1269	1269
q18	7654	7388	7350	7350
q19	854	1158	1121	1121
q20	2038	2096	1940	1940
q21	5353	5005	4578	4578
q22	1134	1102	1072	1072
Total cold run time: 53324 ms
Total hot run time: 52012 ms

@doris-robot
Copy link

doris-robot commented Oct 15, 2025

TPC-DS: Total hot run time: 192799 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ff3256c223eecdbfb192c8226fdf896046562d00, data reload: false

query1	962	411	425	411
query2	6137	1900	1887	1887
query3	8681	205	200	200
query4	33622	23790	23493	23493
query5	3535	591	469	469
query6	305	198	169	169
query7	4203	477	301	301
query8	307	245	230	230
query9	9297	2587	2581	2581
query10	487	319	260	260
query11	18223	15846	15307	15307
query12	167	116	110	110
query13	1571	535	419	419
query14	10840	7534	6824	6824
query15	232	195	182	182
query16	8014	644	526	526
query17	1573	776	619	619
query18	2227	416	326	326
query19	226	192	168	168
query20	132	134	123	123
query21	213	132	114	114
query22	4661	4547	4448	4448
query23	35009	34468	34190	34190
query24	7238	2696	2706	2696
query25	524	490	429	429
query26	839	284	172	172
query27	2118	488	364	364
query28	5622	2236	2203	2203
query29	709	613	463	463
query30	242	196	158	158
query31	1006	943	844	844
query32	96	68	60	60
query33	511	404	313	313
query34	753	860	530	530
query35	813	813	733	733
query36	1027	1083	980	980
query37	109	93	80	80
query38	4038	4108	3999	3999
query39	1549	1491	1479	1479
query40	202	116	101	101
query41	57	54	52	52
query42	131	112	107	107
query43	519	511	492	492
query44	1321	823	820	820
query45	187	184	174	174
query46	881	1056	681	681
query47	1979	1958	1935	1935
query48	406	438	358	358
query49	748	479	396	396
query50	668	692	432	432
query51	7375	7282	7190	7190
query52	109	110	94	94
query53	234	256	191	191
query54	549	538	463	463
query55	78	86	83	83
query56	295	282	275	275
query57	1290	1272	1213	1213
query58	235	229	230	229
query59	3088	3182	3058	3058
query60	305	293	280	280
query61	115	109	133	109
query62	826	741	716	716
query63	241	201	195	195
query64	3710	986	655	655
query65	3399	3301	3341	3301
query66	816	417	314	314
query67	16350	15902	15614	15614
query68	7916	807	534	534
query69	474	299	281	281
query70	1173	1138	1104	1104
query71	374	297	264	264
query72	5806	3733	3770	3733
query73	625	732	347	347
query74	10592	9248	9216	9216
query75	3214	3154	2667	2667
query76	3132	1172	774	774
query77	668	377	284	284
query78	10386	10505	9717	9717
query79	3803	835	572	572
query80	767	516	444	444
query81	499	259	224	224
query82	579	117	89	89
query83	173	164	140	140
query84	282	106	97	97
query85	787	369	305	305
query86	367	337	290	290
query87	4387	4376	4246	4246
query88	4178	2499	2403	2403
query89	414	337	294	294
query90	1840	188	188	188
query91	137	151	107	107
query92	68	56	52	52
query93	1948	865	538	538
query94	681	404	311	311
query95	347	281	271	271
query96	485	607	292	292
query97	3153	3282	3117	3117
query98	218	218	196	196
query99	1561	1425	1314	1314
Total cold run time: 294966 ms
Total hot run time: 192799 ms

@doris-robot
Copy link

doris-robot commented Oct 15, 2025

ClickBench: Total hot run time: 29.23 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ff3256c223eecdbfb192c8226fdf896046562d00, data reload: false

query1	0.03	0.03	0.02
query2	0.06	0.04	0.03
query3	0.23	0.07	0.06
query4	1.62	0.10	0.10
query5	0.52	0.51	0.53
query6	1.14	0.73	0.71
query7	0.03	0.03	0.02
query8	0.06	0.03	0.04
query9	0.56	0.50	0.52
query10	0.55	0.55	0.57
query11	0.14	0.11	0.10
query12	0.14	0.10	0.11
query13	0.62	0.60	0.59
query14	0.78	0.78	0.81
query15	0.85	0.83	0.82
query16	0.38	0.41	0.39
query17	1.00	1.00	0.99
query18	0.25	0.24	0.23
query19	1.96	1.82	1.78
query20	0.02	0.02	0.01
query21	15.41	0.99	0.57
query22	0.74	0.73	0.76
query23	15.08	1.47	0.62
query24	3.25	1.39	1.30
query25	0.13	0.05	0.07
query26	0.37	0.17	0.14
query27	0.06	0.06	0.05
query28	13.74	1.01	0.43
query29	12.56	3.90	3.23
query30	0.25	0.10	0.07
query31	2.82	0.59	0.40
query32	3.23	0.55	0.46
query33	2.98	3.03	3.10
query34	16.71	5.12	4.52
query35	4.52	4.53	4.56
query36	0.65	0.50	0.47
query37	0.08	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.07	0.03	0.02
query42	0.03	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 103.89 s
Total hot run time: 29.23 s

@wenzhenghu
Copy link
Contributor Author

wenzhenghu commented Oct 16, 2025

run NonConcurrentRegression

@wenzhenghu
Copy link
Contributor Author

wenzhenghu commented Oct 16, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 16, 2025

TPC-H: Total hot run time: 32596 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ff3256c223eecdbfb192c8226fdf896046562d00, data reload: false

------ Round 1 ----------------------------------
q1	17600	5508	5408	5408
q2	2022	373	274	274
q3	12012	1241	747	747
q4	10288	870	451	451
q5	9112	2367	2145	2145
q6	201	168	133	133
q7	902	741	614	614
q8	9335	1433	1117	1117
q9	5147	4932	4926	4926
q10	6739	2260	1804	1804
q11	460	273	256	256
q12	339	351	205	205
q13	18260	3595	3061	3061
q14	239	229	217	217
q15	531	470	459	459
q16	441	435	368	368
q17	577	860	359	359
q18	7201	6331	6346	6331
q19	1208	951	530	530
q20	331	341	197	197
q21	2731	2136	1994	1994
q22	1084	1046	1000	1000
Total cold run time: 106760 ms
Total hot run time: 32596 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5500	5511	5490	5490
q2	235	325	233	233
q3	2249	2620	2303	2303
q4	1380	1842	1407	1407
q5	4370	4986	4966	4966
q6	167	165	129	129
q7	2069	1956	1796	1796
q8	2634	2798	2711	2711
q9	7318	7264	7198	7198
q10	3025	3270	2767	2767
q11	580	507	497	497
q12	663	781	583	583
q13	3380	3816	3202	3202
q14	272	320	271	271
q15	525	479	475	475
q16	433	477	439	439
q17	1213	1705	1273	1273
q18	7674	7573	7320	7320
q19	783	1169	1076	1076
q20	2008	2044	1901	1901
q21	5268	4851	4387	4387
q22	1120	1121	1033	1033
Total cold run time: 52866 ms
Total hot run time: 51457 ms

@doris-robot
Copy link

doris-robot commented Oct 16, 2025

TPC-DS: Total hot run time: 192584 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ff3256c223eecdbfb192c8226fdf896046562d00, data reload: false

query1	948	404	425	404
query2	6216	1949	1840	1840
query3	8674	192	192	192
query4	33725	24167	23501	23501
query5	3611	607	462	462
query6	293	198	180	180
query7	4205	493	320	320
query8	319	257	244	244
query9	9350	2577	2514	2514
query10	469	321	257	257
query11	18621	15747	15179	15179
query12	156	104	107	104
query13	1568	536	402	402
query14	9804	7151	7483	7151
query15	240	195	184	184
query16	8028	668	492	492
query17	1593	786	596	596
query18	2168	420	322	322
query19	238	184	175	175
query20	130	129	134	129
query21	211	128	116	116
query22	4646	4526	4575	4526
query23	35009	34756	34031	34031
query24	7684	2664	2681	2664
query25	521	464	427	427
query26	835	276	182	182
query27	2210	477	350	350
query28	4950	2209	2172	2172
query29	688	579	446	446
query30	232	196	168	168
query31	1021	941	838	838
query32	96	56	58	56
query33	498	362	309	309
query34	740	861	531	531
query35	778	817	744	744
query36	1012	1069	994	994
query37	108	94	69	69
query38	3985	3976	3974	3974
query39	1532	1474	1499	1474
query40	212	115	106	106
query41	55	49	52	49
query42	132	107	113	107
query43	515	516	490	490
query44	1361	835	821	821
query45	196	182	173	173
query46	905	1069	673	673
query47	2011	2017	1948	1948
query48	404	417	343	343
query49	749	496	402	402
query50	682	701	448	448
query51	7466	7338	7189	7189
query52	109	103	94	94
query53	238	274	198	198
query54	557	569	493	493
query55	84	81	81	81
query56	292	286	266	266
query57	1265	1303	1206	1206
query58	247	228	240	228
query59	3125	3270	2992	2992
query60	311	314	300	300
query61	140	142	147	142
query62	794	766	723	723
query63	267	205	198	198
query64	3725	1015	643	643
query65	3383	3294	3298	3294
query66	857	416	304	304
query67	16407	15684	15671	15671
query68	5173	813	534	534
query69	485	298	265	265
query70	1194	1137	1120	1120
query71	372	292	266	266
query72	5712	3741	3945	3741
query73	625	742	348	348
query74	10196	9165	9175	9165
query75	3213	3180	2683	2683
query76	2981	1189	776	776
query77	526	368	282	282
query78	10320	10476	9529	9529
query79	2352	889	604	604
query80	707	518	430	430
query81	487	264	226	226
query82	185	124	96	96
query83	170	169	152	152
query84	242	111	86	86
query85	810	429	366	366
query86	378	330	295	295
query87	4436	4320	4220	4220
query88	4528	2430	2396	2396
query89	390	335	294	294
query90	1874	189	186	186
query91	134	145	109	109
query92	74	55	51	51
query93	2626	906	531	531
query94	658	424	288	288
query95	348	279	269	269
query96	487	604	279	279
query97	3117	3243	3179	3179
query98	216	210	213	210
query99	1330	1414	1307	1307
Total cold run time: 289947 ms
Total hot run time: 192584 ms

@wenzhenghu wenzhenghu changed the title [draft][improvement](memory) Add configurable threshold for small memory task protection [improvement](memory) Add configurable threshold for small memory task protection Oct 16, 2025
@wenzhenghu wenzhenghu changed the title [improvement](memory) Add configurable threshold for small memory task protection [draft][improvement](memory) Add configurable threshold for small memory task protection Oct 16, 2025
@wenzhenghu
Copy link
Contributor Author

wenzhenghu commented Oct 16, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 16, 2025

TPC-H: Total hot run time: 32757 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ff3256c223eecdbfb192c8226fdf896046562d00, data reload: false

------ Round 1 ----------------------------------
q1	17587	5597	5489	5489
q2	2042	281	155	155
q3	10457	1246	722	722
q4	10214	857	456	456
q5	7658	2354	2117	2117
q6	181	166	137	137
q7	902	736	616	616
q8	9340	1438	1199	1199
q9	5293	4936	4924	4924
q10	6766	2285	1813	1813
q11	482	285	268	268
q12	349	373	210	210
q13	17780	3600	3044	3044
q14	225	224	209	209
q15	527	459	459	459
q16	413	417	375	375
q17	577	864	366	366
q18	7013	6538	6427	6427
q19	1217	963	529	529
q20	328	341	210	210
q21	2819	2165	2016	2016
q22	1066	1017	1016	1016
Total cold run time: 103236 ms
Total hot run time: 32757 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5577	5488	5510	5488
q2	234	331	229	229
q3	2224	2599	2272	2272
q4	1325	1750	1399	1399
q5	4350	4845	4679	4679
q6	160	154	125	125
q7	1939	1851	1712	1712
q8	2485	2659	2572	2572
q9	6872	6849	6792	6792
q10	2890	3156	2632	2632
q11	570	494	487	487
q12	627	695	545	545
q13	3251	3612	3014	3014
q14	278	280	264	264
q15	518	461	457	457
q16	430	452	432	432
q17	1172	1700	1217	1217
q18	7163	7060	6872	6872
q19	740	984	1096	984
q20	1952	1993	1819	1819
q21	4983	4736	4500	4500
q22	1067	1042	996	996
Total cold run time: 50807 ms
Total hot run time: 49487 ms

@doris-robot
Copy link

doris-robot commented Oct 16, 2025

TPC-DS: Total hot run time: 190068 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ff3256c223eecdbfb192c8226fdf896046562d00, data reload: false

query1	974	386	380	380
query2	6553	1900	1911	1900
query3	6716	216	216	216
query4	33881	23880	23586	23586
query5	4824	635	451	451
query6	279	198	178	178
query7	4638	491	314	314
query8	315	259	241	241
query9	9558	2592	2595	2592
query10	492	326	265	265
query11	18180	15474	15265	15265
query12	159	114	109	109
query13	1674	525	420	420
query14	10744	6660	7290	6660
query15	242	193	189	189
query16	7962	646	501	501
query17	1578	747	570	570
query18	2116	405	315	315
query19	226	183	166	166
query20	123	113	113	113
query21	214	127	108	108
query22	4348	4206	4284	4206
query23	34310	33279	34069	33279
query24	7191	2726	2663	2663
query25	526	476	423	423
query26	1208	287	177	177
query27	1980	488	337	337
query28	5000	2171	2109	2109
query29	791	575	449	449
query30	257	186	158	158
query31	994	876	833	833
query32	97	68	64	64
query33	526	377	320	320
query34	757	824	519	519
query35	761	797	727	727
query36	990	1047	978	978
query37	112	101	70	70
query38	3952	3885	3850	3850
query39	1526	1425	1461	1425
query40	216	118	104	104
query41	53	52	51	51
query42	117	106	104	104
query43	498	505	479	479
query44	1295	796	799	796
query45	189	178	173	173
query46	870	1031	662	662
query47	1917	1920	1869	1869
query48	408	430	370	370
query49	812	508	414	414
query50	680	676	422	422
query51	7207	7166	7032	7032
query52	103	102	96	96
query53	248	257	188	188
query54	555	562	463	463
query55	82	84	82	82
query56	279	263	258	258
query57	1239	1241	1137	1137
query58	231	215	224	215
query59	3020	2999	2891	2891
query60	293	281	272	272
query61	141	114	116	114
query62	796	718	673	673
query63	227	192	189	189
query64	4599	980	636	636
query65	3321	3244	3217	3217
query66	1064	431	304	304
query67	15953	15780	15436	15436
query68	7688	804	530	530
query69	472	312	267	267
query70	1240	1178	1114	1114
query71	377	299	264	264
query72	5633	3713	3723	3713
query73	630	742	347	347
query74	10000	9429	9225	9225
query75	3222	3137	2652	2652
query76	3256	1189	807	807
query77	458	355	288	288
query78	10272	10347	9612	9612
query79	3533	909	600	600
query80	685	538	444	444
query81	524	258	227	227
query82	651	126	91	91
query83	164	163	146	146
query84	240	107	79	79
query85	798	396	321	321
query86	402	333	315	315
query87	4336	4377	4229	4229
query88	5401	2402	2378	2378
query89	399	336	298	298
query90	1963	191	186	186
query91	135	138	109	109
query92	62	56	54	54
query93	2255	899	548	548
query94	703	413	304	304
query95	342	275	267	267
query96	483	604	282	282
query97	3151	3236	3107	3107
query98	236	218	202	202
query99	1380	1459	1289	1289
Total cold run time: 294278 ms
Total hot run time: 190068 ms

@doris-robot
Copy link

doris-robot commented Oct 16, 2025

ClickBench: Total hot run time: 28.33 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ff3256c223eecdbfb192c8226fdf896046562d00, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.63	0.11	0.11
query5	0.52	0.49	0.52
query6	1.13	0.74	0.73
query7	0.03	0.02	0.02
query8	0.05	0.04	0.03
query9	0.56	0.52	0.50
query10	0.55	0.55	0.55
query11	0.14	0.10	0.10
query12	0.14	0.10	0.11
query13	0.63	0.59	0.59
query14	0.79	0.80	0.80
query15	0.87	0.83	0.83
query16	0.39	0.39	0.40
query17	1.02	1.01	1.06
query18	0.24	0.22	0.22
query19	1.98	1.85	1.82
query20	0.01	0.01	0.01
query21	15.37	0.92	0.58
query22	0.74	0.84	0.59
query23	15.15	1.46	0.61
query24	3.44	0.36	1.92
query25	0.12	0.23	0.07
query26	0.34	0.15	0.14
query27	0.06	0.05	0.05
query28	13.42	1.01	0.44
query29	12.64	3.90	3.21
query30	0.25	0.08	0.06
query31	2.83	0.60	0.37
query32	3.22	0.53	0.46
query33	3.02	3.05	3.02
query34	16.63	5.21	4.61
query35	4.64	4.57	4.52
query36	0.63	0.50	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104 s
Total hot run time: 28.33 s

@wenzhenghu
Copy link
Contributor Author

wenzhenghu commented Oct 16, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 16, 2025

TPC-H: Total hot run time: 32621 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0abd4921544ed946f4221c92b69eec6864ac295b, data reload: false

------ Round 1 ----------------------------------
q1	17819	5541	5463	5463
q2	2055	394	296	296
q3	12344	1227	736	736
q4	10308	869	438	438
q5	9249	2355	2092	2092
q6	186	171	133	133
q7	916	755	614	614
q8	9342	1417	1125	1125
q9	5244	4925	4871	4871
q10	6758	2282	1842	1842
q11	483	271	268	268
q12	328	347	205	205
q13	17804	3600	3010	3010
q14	217	226	208	208
q15	524	470	460	460
q16	417	417	371	371
q17	573	857	360	360
q18	6845	6409	6413	6409
q19	1232	942	536	536
q20	336	344	210	210
q21	2792	2132	1999	1999
q22	1065	1027	975	975
Total cold run time: 106837 ms
Total hot run time: 32621 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5527	5497	5494	5494
q2	231	334	240	240
q3	2291	2610	2326	2326
q4	1345	1800	1359	1359
q5	4400	4972	4959	4959
q6	167	164	131	131
q7	2096	1982	1816	1816
q8	2623	2834	2708	2708
q9	7220	7182	7219	7182
q10	3020	3297	2737	2737
q11	596	522	513	513
q12	670	785	608	608
q13	3372	3848	3215	3215
q14	289	302	285	285
q15	522	479	463	463
q16	434	493	427	427
q17	1233	1742	1234	1234
q18	7774	7521	7353	7353
q19	782	1167	1048	1048
q20	2011	2058	1948	1948
q21	5307	4792	4597	4597
q22	1100	1049	1052	1049
Total cold run time: 53010 ms
Total hot run time: 51692 ms

@doris-robot
Copy link

doris-robot commented Oct 16, 2025

TPC-DS: Total hot run time: 192385 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0abd4921544ed946f4221c92b69eec6864ac295b, data reload: false

query1	960	398	413	398
query2	6153	1897	1847	1847
query3	8708	204	194	194
query4	33810	24103	23677	23677
query5	3801	632	468	468
query6	279	197	191	191
query7	4215	485	316	316
query8	302	245	229	229
query9	9139	2589	2586	2586
query10	471	318	250	250
query11	18039	15447	15216	15216
query12	150	110	109	109
query13	1575	543	418	418
query14	9399	6675	6670	6670
query15	227	192	187	187
query16	7995	672	516	516
query17	1597	774	613	613
query18	2135	417	329	329
query19	224	194	168	168
query20	138	126	126	126
query21	215	127	107	107
query22	4618	4673	4471	4471
query23	35285	34672	33985	33985
query24	7722	2700	2708	2700
query25	520	457	401	401
query26	791	280	169	169
query27	2349	451	353	353
query28	5716	2210	2167	2167
query29	693	589	437	437
query30	249	189	153	153
query31	983	959	831	831
query32	95	56	57	56
query33	488	382	325	325
query34	753	852	524	524
query35	819	810	728	728
query36	1022	1049	942	942
query37	105	93	69	69
query38	4071	4080	3988	3988
query39	1569	1487	1491	1487
query40	210	114	105	105
query41	53	52	47	47
query42	121	103	105	103
query43	497	527	474	474
query44	1413	858	830	830
query45	186	174	176	174
query46	881	1056	665	665
query47	1995	1974	1950	1950
query48	408	438	372	372
query49	802	486	401	401
query50	665	684	424	424
query51	7472	7302	7307	7302
query52	107	100	92	92
query53	232	259	186	186
query54	545	553	469	469
query55	85	83	83	83
query56	288	275	265	265
query57	1278	1266	1218	1218
query58	231	229	230	229
query59	3106	3160	3010	3010
query60	302	300	284	284
query61	115	113	117	113
query62	829	751	697	697
query63	232	195	198	195
query64	3857	1131	763	763
query65	3396	3327	3352	3327
query66	1064	404	306	306
query67	16569	15704	15684	15684
query68	7941	800	526	526
query69	495	296	268	268
query70	1174	1134	1149	1134
query71	365	292	265	265
query72	5137	3749	3925	3749
query73	632	738	349	349
query74	10578	9253	9164	9164
query75	3215	3122	2638	2638
query76	3013	1184	763	763
query77	485	373	284	284
query78	10372	10455	9601	9601
query79	2539	886	602	602
query80	587	528	438	438
query81	488	262	222	222
query82	201	122	86	86
query83	164	160	148	148
query84	248	93	79	79
query85	774	366	297	297
query86	391	320	298	298
query87	4283	4392	4231	4231
query88	5339	2404	2390	2390
query89	390	335	287	287
query90	1868	191	194	191
query91	139	143	115	115
query92	62	54	51	51
query93	1922	876	532	532
query94	639	424	305	305
query95	351	281	272	272
query96	484	601	284	284
query97	3149	3222	3169	3169
query98	236	207	194	194
query99	1320	1396	1284	1284
Total cold run time: 293300 ms
Total hot run time: 192385 ms

@doris-robot
Copy link

doris-robot commented Oct 16, 2025

ClickBench: Total hot run time: 28.65 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0abd4921544ed946f4221c92b69eec6864ac295b, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.02	0.03
query3	0.24	0.06	0.06
query4	1.63	0.10	0.10
query5	0.51	0.51	0.52
query6	1.14	0.74	0.72
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.61	0.51	0.53
query10	0.55	0.54	0.54
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.60	0.59
query14	0.76	0.83	0.79
query15	0.84	0.83	0.82
query16	0.39	0.40	0.39
query17	1.08	1.06	1.07
query18	0.22	0.22	0.22
query19	1.87	1.92	1.83
query20	0.01	0.01	0.01
query21	15.41	0.93	0.58
query22	0.73	0.80	0.67
query23	15.10	1.46	0.62
query24	3.00	0.63	1.30
query25	0.22	0.06	0.07
query26	0.42	0.15	0.13
query27	0.05	0.06	0.05
query28	12.95	1.06	0.43
query29	12.56	3.89	3.19
query30	0.27	0.09	0.07
query31	2.81	0.59	0.38
query32	3.23	0.54	0.46
query33	2.99	3.03	3.03
query34	16.49	5.19	4.53
query35	4.56	4.58	4.60
query36	0.65	0.49	0.46
query37	0.08	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 102.84 s
Total hot run time: 28.65 s

@wenzhenghu
Copy link
Contributor Author

wenzhenghu commented Oct 16, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 16, 2025

TPC-H: Total hot run time: 32854 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b0c609bfadeb61d8789d28456c34727c76c504ca, data reload: false

------ Round 1 ----------------------------------
q1	17592	5503	5476	5476
q2	2029	393	285	285
q3	12257	1270	752	752
q4	10246	871	461	461
q5	8677	2415	2166	2166
q6	187	166	137	137
q7	918	754	617	617
q8	9346	1425	1156	1156
q9	5290	4956	4932	4932
q10	6744	2281	1821	1821
q11	486	290	272	272
q12	334	362	217	217
q13	17782	3627	3023	3023
q14	216	221	209	209
q15	520	477	472	472
q16	423	432	369	369
q17	583	876	354	354
q18	7029	6371	6528	6371
q19	1224	969	537	537
q20	323	336	207	207
q21	2817	2268	2014	2014
q22	1073	1049	1006	1006
Total cold run time: 106096 ms
Total hot run time: 32854 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5527	5526	5505	5505
q2	235	333	230	230
q3	2253	2644	2323	2323
q4	1419	1770	1329	1329
q5	4384	5025	4970	4970
q6	175	164	133	133
q7	2061	1972	1816	1816
q8	2694	2822	2742	2742
q9	7236	7232	7229	7229
q10	3032	3327	2697	2697
q11	574	515	522	515
q12	676	816	571	571
q13	3406	3828	3183	3183
q14	301	305	272	272
q15	521	475	466	466
q16	440	494	429	429
q17	1243	1734	1278	1278
q18	7716	7398	7352	7352
q19	777	1042	1130	1042
q20	2009	2084	1902	1902
q21	5400	4916	4727	4727
q22	1071	1083	1029	1029
Total cold run time: 53150 ms
Total hot run time: 51740 ms

@wenzhenghu wenzhenghu changed the title [draft][improvement](memory) Add configurable threshold for small memory task protection [draft][enhancement](memory) Add configurable threshold for small memory task protection Oct 16, 2025
@doris-robot
Copy link

doris-robot commented Oct 16, 2025

TPC-DS: Total hot run time: 192853 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b0c609bfadeb61d8789d28456c34727c76c504ca, data reload: false

query1	948	403	406	403
query2	6216	1950	1890	1890
query3	8697	206	202	202
query4	33601	24451	23550	23550
query5	4309	583	457	457
query6	302	199	180	180
query7	4204	485	321	321
query8	308	281	257	257
query9	9324	2612	2602	2602
query10	493	339	271	271
query11	18041	15367	15489	15367
query12	153	111	108	108
query13	1549	537	429	429
query14	9688	6740	6781	6740
query15	255	191	182	182
query16	8035	703	528	528
query17	1538	746	602	602
query18	2154	441	334	334
query19	204	193	177	177
query20	128	130	121	121
query21	212	135	117	117
query22	4582	4634	4475	4475
query23	35307	34575	34062	34062
query24	8410	2754	2751	2751
query25	538	467	426	426
query26	977	286	186	186
query27	2646	467	362	362
query28	5750	2230	2193	2193
query29	777	581	434	434
query30	244	191	160	160
query31	1021	942	837	837
query32	93	58	60	58
query33	503	381	346	346
query34	765	889	512	512
query35	801	818	739	739
query36	1021	1044	941	941
query37	105	95	71	71
query38	4057	4004	3991	3991
query39	1550	1477	1473	1473
query40	210	123	114	114
query41	51	50	48	48
query42	120	102	99	99
query43	520	525	480	480
query44	1374	832	808	808
query45	186	179	176	176
query46	877	1063	688	688
query47	1930	1928	1911	1911
query48	432	430	356	356
query49	755	508	414	414
query50	662	702	456	456
query51	7386	7204	7238	7204
query52	104	107	91	91
query53	238	259	202	202
query54	556	559	485	485
query55	79	81	81	81
query56	275	276	273	273
query57	1274	1301	1209	1209
query58	247	240	226	226
query59	3177	3207	3046	3046
query60	301	303	274	274
query61	118	116	137	116
query62	804	755	693	693
query63	240	200	206	200
query64	4525	1071	758	758
query65	3382	3315	3302	3302
query66	1023	431	321	321
query67	16142	15750	15567	15567
query68	7002	821	540	540
query69	495	325	288	288
query70	1212	1156	1142	1142
query71	367	289	279	279
query72	5659	3793	3686	3686
query73	633	733	346	346
query74	10389	9142	9262	9142
query75	3185	3124	2659	2659
query76	2976	1175	775	775
query77	464	373	284	284
query78	10402	10483	9718	9718
query79	3011	883	610	610
query80	726	524	474	474
query81	506	255	224	224
query82	396	126	88	88
query83	172	162	153	153
query84	293	102	78	78
query85	791	350	339	339
query86	390	310	276	276
query87	4358	4327	4232	4232
query88	5043	2403	2388	2388
query89	393	346	296	296
query90	1826	191	192	191
query91	139	135	109	109
query92	62	57	50	50
query93	2450	892	529	529
query94	696	398	329	329
query95	354	286	272	272
query96	479	605	279	279
query97	3183	3242	3136	3136
query98	227	219	202	202
query99	1523	1419	1286	1286
Total cold run time: 296296 ms
Total hot run time: 192853 ms

@doris-robot
Copy link

doris-robot commented Oct 16, 2025

ClickBench: Total hot run time: 28.31 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b0c609bfadeb61d8789d28456c34727c76c504ca, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.24	0.07	0.07
query4	1.63	0.11	0.10
query5	0.53	0.50	0.51
query6	1.12	0.72	0.72
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.57	0.50	0.52
query10	0.56	0.55	0.56
query11	0.14	0.10	0.10
query12	0.13	0.11	0.11
query13	0.63	0.60	0.59
query14	0.79	0.82	0.80
query15	0.85	0.83	0.83
query16	0.39	0.42	0.40
query17	1.04	1.10	1.06
query18	0.24	0.22	0.22
query19	1.92	1.77	1.85
query20	0.02	0.01	0.01
query21	15.38	0.95	0.60
query22	0.73	0.81	0.58
query23	15.21	1.39	0.61
query24	3.55	0.47	0.30
query25	0.15	0.07	0.06
query26	0.40	0.15	0.15
query27	0.07	0.06	0.05
query28	12.69	1.07	0.44
query29	12.55	3.92	3.30
query30	0.26	0.09	0.07
query31	2.83	0.61	0.38
query32	3.23	0.53	0.46
query33	3.03	3.02	3.07
query34	16.52	5.19	4.53
query35	4.55	4.55	4.54
query36	0.65	0.49	0.49
query37	0.09	0.06	0.05
query38	0.04	0.03	0.04
query39	0.04	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.02
Total cold run time: 103.25 s
Total hot run time: 28.31 s

@wenzhenghu
Copy link
Contributor Author

wenzhenghu commented Oct 17, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 17, 2025

TPC-H: Total hot run time: 33124 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b7a4d6ba2ad3144c320d68425204797b39870180, data reload: false

------ Round 1 ----------------------------------
q1	17959	5580	5540	5540
q2	2031	435	290	290
q3	12078	1263	776	776
q4	10357	871	454	454
q5	9383	2433	2149	2149
q6	197	167	133	133
q7	896	749	619	619
q8	9354	1479	1194	1194
q9	5277	5106	5013	5013
q10	6793	2286	1838	1838
q11	492	308	272	272
q12	341	359	221	221
q13	17765	3584	2998	2998
q14	248	240	210	210
q15	535	459	469	459
q16	429	438	387	387
q17	619	877	375	375
q18	6939	6442	6442	6442
q19	1532	984	574	574
q20	327	351	206	206
q21	2844	2171	1980	1980
q22	1085	1032	994	994
Total cold run time: 107481 ms
Total hot run time: 33124 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5637	5534	5523	5523
q2	242	330	234	234
q3	2221	2630	2335	2335
q4	1378	1791	1355	1355
q5	4408	5153	5026	5026
q6	167	162	127	127
q7	2126	1964	1854	1854
q8	2658	2802	2730	2730
q9	7451	7326	7319	7319
q10	3043	3251	2756	2756
q11	579	509	511	509
q12	724	775	592	592
q13	3439	3817	3194	3194
q14	281	291	276	276
q15	535	482	480	480
q16	443	495	443	443
q17	1239	1765	1258	1258
q18	7617	7497	7389	7389
q19	888	1089	1123	1089
q20	2049	2066	1921	1921
q21	5512	5040	4581	4581
q22	1078	1071	1027	1027
Total cold run time: 53715 ms
Total hot run time: 52018 ms

@doris-robot
Copy link

doris-robot commented Oct 17, 2025

TPC-DS: Total hot run time: 194308 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b7a4d6ba2ad3144c320d68425204797b39870180, data reload: false

query1	939	462	409	409
query2	6243	1990	1964	1964
query3	8688	204	200	200
query4	33584	23926	23538	23538
query5	3764	613	488	488
query6	301	214	179	179
query7	4209	499	317	317
query8	302	250	239	239
query9	9358	2625	2604	2604
query10	480	341	286	286
query11	17957	15511	15410	15410
query12	162	114	112	112
query13	1555	553	422	422
query14	9342	7303	7766	7303
query15	256	204	190	190
query16	8144	653	475	475
query17	1581	804	614	614
query18	2221	434	343	343
query19	247	193	174	174
query20	137	125	119	119
query21	210	130	112	112
query22	4528	4663	4343	4343
query23	35021	33999	34756	33999
query24	7671	2771	2774	2771
query25	556	520	437	437
query26	798	303	182	182
query27	2127	507	369	369
query28	5047	2249	2210	2210
query29	735	607	465	465
query30	246	190	171	171
query31	991	904	852	852
query32	88	67	66	66
query33	524	385	340	340
query34	776	871	532	532
query35	817	849	734	734
query36	1031	1056	979	979
query37	123	101	73	73
query38	4114	4017	4000	4000
query39	1522	1503	1515	1503
query40	222	121	111	111
query41	50	51	48	48
query42	125	111	112	111
query43	526	531	506	506
query44	1397	842	845	842
query45	193	183	170	170
query46	908	1087	697	697
query47	1992	2000	1973	1973
query48	421	442	351	351
query49	719	496	431	431
query50	710	700	446	446
query51	7366	7400	7341	7341
query52	103	101	96	96
query53	243	269	195	195
query54	580	578	493	493
query55	84	82	86	82
query56	292	284	253	253
query57	1279	1274	1222	1222
query58	238	226	223	223
query59	3169	3216	3096	3096
query60	297	290	284	284
query61	127	116	126	116
query62	794	767	708	708
query63	235	206	196	196
query64	4051	1023	631	631
query65	3431	3359	3318	3318
query66	817	411	331	331
query67	16611	15850	15715	15715
query68	8404	845	550	550
query69	475	311	274	274
query70	1157	1125	1098	1098
query71	423	305	277	277
query72	5145	3805	3883	3805
query73	651	761	357	357
query74	10244	9374	9175	9175
query75	3839	3162	2730	2730
query76	3498	1196	795	795
query77	764	388	285	285
query78	10364	10423	9725	9725
query79	6374	875	595	595
query80	661	549	481	481
query81	487	262	228	228
query82	236	123	93	93
query83	170	163	146	146
query84	298	106	79	79
query85	749	391	304	304
query86	355	310	304	304
query87	4417	4352	4289	4289
query88	4120	2453	2436	2436
query89	425	337	298	298
query90	2116	198	197	197
query91	133	145	113	113
query92	65	59	53	53
query93	3559	897	543	543
query94	672	418	308	308
query95	355	291	291	291
query96	498	632	289	289
query97	3194	3222	3158	3158
query98	224	201	205	201
query99	1468	1438	1321	1321
Total cold run time: 298915 ms
Total hot run time: 194308 ms

@doris-robot
Copy link

doris-robot commented Oct 17, 2025

ClickBench: Total hot run time: 29.15 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b7a4d6ba2ad3144c320d68425204797b39870180, data reload: false

query1	0.04	0.04	0.02
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.11	0.11
query5	0.53	0.52	0.51
query6	1.13	0.72	0.74
query7	0.02	0.02	0.02
query8	0.05	0.03	0.03
query9	0.56	0.51	0.49
query10	0.56	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.10	0.12
query13	0.62	0.60	0.60
query14	0.78	0.79	0.79
query15	0.86	0.83	0.83
query16	0.41	0.38	0.40
query17	0.99	1.00	1.07
query18	0.23	0.23	0.23
query19	1.85	1.89	1.89
query20	0.01	0.01	0.01
query21	15.40	0.88	0.56
query22	0.74	0.78	0.55
query23	15.24	1.50	0.67
query24	2.94	1.00	1.78
query25	0.16	0.07	0.07
query26	0.33	0.16	0.14
query27	0.05	0.04	0.04
query28	13.35	1.02	0.44
query29	12.60	4.00	3.35
query30	0.24	0.09	0.09
query31	2.82	0.60	0.38
query32	3.24	0.56	0.46
query33	2.99	3.05	3.01
query34	16.75	5.20	4.55
query35	4.64	4.58	4.57
query36	0.63	0.50	0.47
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 103.47 s
Total hot run time: 29.15 s

@wenzhenghu wenzhenghu changed the title [draft][enhancement](memory) Add configurable threshold for small memory task protection [enhancement](memory) Add configurable threshold for small memory task protection Oct 17, 2025
@wenzhenghu
Copy link
Contributor Author

wenzhenghu commented Oct 20, 2025

run buildall

@wenzhenghu wenzhenghu changed the title [enhancement](memory) Add configurable threshold for small memory task protection branch-3.1: [enhancement](memory) Add configurable threshold for small memory task protection Oct 20, 2025
@doris-robot
Copy link

doris-robot commented Oct 20, 2025

TPC-H: Total hot run time: 33159 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 01f5c6a36f82c5fc674862f35cd3df4ffefe7cd1, data reload: false

------ Round 1 ----------------------------------
q1	17582	5628	5497	5497
q2	2027	398	284	284
q3	11807	1306	761	761
q4	10252	896	478	478
q5	8429	2423	2164	2164
q6	193	170	136	136
q7	917	747	623	623
q8	9343	1481	1199	1199
q9	5268	4992	4962	4962
q10	6793	2282	1827	1827
q11	479	290	270	270
q12	342	361	227	227
q13	17790	3619	3087	3087
q14	243	236	224	224
q15	526	461	478	461
q16	458	436	389	389
q17	642	889	393	393
q18	7151	6579	6373	6373
q19	1215	974	550	550
q20	352	355	222	222
q21	3020	2183	2032	2032
q22	1041	1037	1000	1000
Total cold run time: 105870 ms
Total hot run time: 33159 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5584	5523	5551	5523
q2	238	338	239	239
q3	2301	2678	2334	2334
q4	1387	1793	1417	1417
q5	4487	5035	5080	5035
q6	183	169	136	136
q7	2084	1948	1835	1835
q8	2686	2866	2764	2764
q9	7408	7260	7296	7260
q10	3069	3232	2832	2832
q11	594	527	505	505
q12	671	786	614	614
q13	3433	3808	3197	3197
q14	299	311	281	281
q15	533	467	486	467
q16	433	494	452	452
q17	1254	1752	1312	1312
q18	7697	7539	7213	7213
q19	846	1214	1056	1056
q20	2034	2081	1951	1951
q21	5532	5070	4647	4647
q22	1097	1124	1049	1049
Total cold run time: 53850 ms
Total hot run time: 52119 ms

@doris-robot
Copy link

doris-robot commented Oct 20, 2025

TPC-DS: Total hot run time: 195150 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 01f5c6a36f82c5fc674862f35cd3df4ffefe7cd1, data reload: false

query1	948	434	417	417
query2	6244	2090	2078	2078
query3	8695	212	228	212
query4	33602	24229	24063	24063
query5	4353	626	464	464
query6	314	211	192	192
query7	4197	505	333	333
query8	323	248	248	248
query9	9496	2704	2691	2691
query10	511	338	269	269
query11	18302	16160	15353	15353
query12	159	113	109	109
query13	1554	559	435	435
query14	10531	7360	7058	7058
query15	244	195	180	180
query16	7952	654	516	516
query17	1514	752	595	595
query18	2134	412	332	332
query19	228	185	166	166
query20	125	125	129	125
query21	203	124	116	116
query22	4548	4737	4445	4445
query23	35299	34874	34158	34158
query24	7428	2764	2746	2746
query25	532	485	414	414
query26	909	281	181	181
query27	2204	490	379	379
query28	5205	2235	2216	2216
query29	739	635	464	464
query30	240	197	161	161
query31	1008	941	856	856
query32	89	65	62	62
query33	513	374	338	338
query34	763	887	548	548
query35	810	837	734	734
query36	1034	1083	985	985
query37	108	104	73	73
query38	4116	4095	4047	4047
query39	1586	1532	1516	1516
query40	213	130	115	115
query41	55	54	54	54
query42	127	112	120	112
query43	548	539	507	507
query44	1423	858	846	846
query45	198	186	173	173
query46	910	1077	697	697
query47	1971	1996	1989	1989
query48	434	458	358	358
query49	756	514	418	418
query50	693	706	442	442
query51	7291	7407	7261	7261
query52	118	104	97	97
query53	242	273	203	203
query54	623	599	504	504
query55	86	82	89	82
query56	270	281	275	275
query57	1253	1277	1216	1216
query58	241	236	225	225
query59	3244	3326	3060	3060
query60	312	303	276	276
query61	124	123	117	117
query62	821	758	722	722
query63	247	206	199	199
query64	3789	1023	674	674
query65	3417	3381	3312	3312
query66	828	423	339	339
query67	16585	16081	15559	15559
query68	7886	847	571	571
query69	483	309	275	275
query70	1246	1130	1102	1102
query71	389	297	263	263
query72	5661	3772	3927	3772
query73	653	767	371	371
query74	10418	9313	9320	9313
query75	3256	3184	2729	2729
query76	3079	1207	800	800
query77	642	418	299	299
query78	10545	10672	9645	9645
query79	3131	949	630	630
query80	612	544	460	460
query81	511	262	221	221
query82	595	121	92	92
query83	169	167	151	151
query84	240	105	84	84
query85	786	379	328	328
query86	372	317	290	290
query87	4346	4391	4313	4313
query88	5109	2539	2523	2523
query89	425	336	307	307
query90	1825	196	196	196
query91	160	142	116	116
query92	73	56	54	54
query93	1841	949	568	568
query94	685	417	321	321
query95	361	290	276	276
query96	523	619	294	294
query97	3192	3297	3160	3160
query98	217	218	197	197
query99	1578	1451	1322	1322
Total cold run time: 296588 ms
Total hot run time: 195150 ms

@doris-robot
Copy link

doris-robot commented Oct 20, 2025

ClickBench: Total hot run time: 29.25 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 01f5c6a36f82c5fc674862f35cd3df4ffefe7cd1, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.11	0.11
query5	0.55	0.52	0.54
query6	1.12	0.73	0.72
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.59	0.52	0.52
query10	0.56	0.56	0.56
query11	0.15	0.10	0.10
query12	0.14	0.10	0.11
query13	0.62	0.60	0.60
query14	0.78	0.80	0.80
query15	0.85	0.83	0.84
query16	0.40	0.39	0.38
query17	1.06	1.10	1.06
query18	0.24	0.23	0.24
query19	2.00	1.88	1.91
query20	0.02	0.03	0.02
query21	15.36	0.94	0.58
query22	0.74	0.77	0.73
query23	15.08	1.41	0.52
query24	3.32	1.77	0.96
query25	0.21	0.09	0.16
query26	0.20	0.15	0.14
query27	0.04	0.05	0.05
query28	13.37	0.93	0.44
query29	12.59	3.93	3.26
query30	0.25	0.08	0.06
query31	2.84	0.64	0.38
query32	3.22	0.53	0.46
query33	3.06	3.04	3.07
query34	16.71	5.19	4.57
query35	4.59	4.60	4.56
query36	0.62	0.50	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.16	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.8 s
Total hot run time: 29.25 s

@wenzhenghu
Copy link
Contributor Author

wenzhenghu commented Oct 21, 2025

run buildall

1 similar comment
@wenzhenghu
Copy link
Contributor Author

wenzhenghu commented Oct 21, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 21, 2025

TPC-H: Total hot run time: 33191 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 01f5c6a36f82c5fc674862f35cd3df4ffefe7cd1, data reload: false

------ Round 1 ----------------------------------
q1	17592	5578	5717	5578
q2	2028	416	280	280
q3	11937	1247	743	743
q4	10293	869	448	448
q5	9339	2569	2163	2163
q6	185	163	130	130
q7	889	746	605	605
q8	9339	1450	1132	1132
q9	5223	5050	4964	4964
q10	6789	2336	1812	1812
q11	456	290	264	264
q12	342	364	208	208
q13	17777	3606	3051	3051
q14	232	222	218	218
q15	524	465	472	465
q16	433	434	378	378
q17	616	871	387	387
q18	6958	6595	6577	6577
q19	1234	969	542	542
q20	346	352	225	225
q21	3133	2205	2016	2016
q22	1100	1026	1005	1005
Total cold run time: 106765 ms
Total hot run time: 33191 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5551	5506	5532	5506
q2	240	336	241	241
q3	2285	2689	2371	2371
q4	1314	1846	1356	1356
q5	4480	5105	5070	5070
q6	174	169	132	132
q7	2065	1968	1840	1840
q8	2661	2905	2743	2743
q9	7350	7320	7310	7310
q10	3021	3228	2822	2822
q11	583	515	499	499
q12	683	789	628	628
q13	3451	3761	3184	3184
q14	300	295	281	281
q15	516	479	471	471
q16	435	505	434	434
q17	1231	1758	1246	1246
q18	7674	7813	7321	7321
q19	819	1140	1110	1110
q20	2012	2118	1956	1956
q21	5322	4905	4614	4614
q22	1120	1127	1050	1050
Total cold run time: 53287 ms
Total hot run time: 52185 ms

@doris-robot
Copy link

doris-robot commented Oct 21, 2025

TPC-DS: Total hot run time: 194130 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 01f5c6a36f82c5fc674862f35cd3df4ffefe7cd1, data reload: false

query1	937	405	434	405
query2	6113	2041	1927	1927
query3	8687	207	201	201
query4	33929	24010	23703	23703
query5	3729	604	455	455
query6	307	201	169	169
query7	4214	503	326	326
query8	308	261	226	226
query9	9385	2644	2640	2640
query10	481	326	270	270
query11	18342	15610	15319	15319
query12	164	116	108	108
query13	1572	568	438	438
query14	9500	6680	7123	6680
query15	239	198	185	185
query16	8082	679	481	481
query17	1559	804	612	612
query18	2129	430	339	339
query19	237	215	177	177
query20	133	126	126	126
query21	211	129	115	115
query22	4612	4745	4574	4574
query23	36301	34997	34613	34613
query24	7424	2747	2745	2745
query25	510	491	436	436
query26	1111	284	181	181
query27	2137	497	370	370
query28	5306	2240	2209	2209
query29	600	595	471	471
query30	239	194	178	178
query31	1025	923	837	837
query32	90	68	86	68
query33	502	404	319	319
query34	756	864	517	517
query35	808	829	741	741
query36	1036	1072	939	939
query37	105	96	68	68
query38	4048	4037	4004	4004
query39	1613	1507	1483	1483
query40	212	129	109	109
query41	52	49	48	48
query42	129	114	111	111
query43	523	536	507	507
query44	1334	861	856	856
query45	191	184	175	175
query46	895	1076	683	683
query47	2003	2005	1963	1963
query48	414	438	359	359
query49	787	517	416	416
query50	703	692	450	450
query51	7396	7241	7112	7112
query52	105	101	94	94
query53	242	275	190	190
query54	559	558	487	487
query55	81	79	83	79
query56	280	285	272	272
query57	1275	1297	1234	1234
query58	239	221	217	217
query59	3068	3358	3102	3102
query60	300	288	281	281
query61	119	119	113	113
query62	820	761	701	701
query63	225	197	197	197
query64	4484	1045	644	644
query65	3379	3346	3279	3279
query66	968	444	312	312
query67	16469	16138	15830	15830
query68	7490	852	555	555
query69	485	319	274	274
query70	1249	1131	1077	1077
query71	366	300	262	262
query72	5606	3634	3705	3634
query73	650	763	362	362
query74	10545	9258	9195	9195
query75	3250	3176	2727	2727
query76	3277	1175	757	757
query77	612	388	278	278
query78	10421	10476	9764	9764
query79	3900	874	610	610
query80	733	529	441	441
query81	516	256	222	222
query82	577	118	97	97
query83	179	167	155	155
query84	286	112	90	90
query85	789	361	306	306
query86	382	318	302	302
query87	4282	4339	4301	4301
query88	5338	2485	2459	2459
query89	407	339	294	294
query90	1759	191	188	188
query91	136	152	120	120
query92	59	54	52	52
query93	2311	899	546	546
query94	670	395	318	318
query95	351	284	272	272
query96	503	628	293	293
query97	3179	3258	3158	3158
query98	236	213	204	204
query99	1527	1408	1341	1341
Total cold run time: 297804 ms
Total hot run time: 194130 ms

@doris-robot
Copy link

doris-robot commented Oct 21, 2025

ClickBench: Total hot run time: 28.95 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 01f5c6a36f82c5fc674862f35cd3df4ffefe7cd1, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.04	0.03
query3	0.24	0.07	0.06
query4	1.62	0.11	0.11
query5	0.51	0.55	0.52
query6	1.12	0.75	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.58	0.50	0.52
query10	0.55	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.63	0.62	0.61
query14	0.79	0.85	0.79
query15	0.85	0.84	0.83
query16	0.40	0.39	0.38
query17	1.08	1.07	1.08
query18	0.23	0.23	0.23
query19	1.89	1.89	1.93
query20	0.02	0.02	0.01
query21	15.37	0.89	0.60
query22	0.75	0.84	0.60
query23	15.09	1.43	0.57
query24	3.34	0.74	1.91
query25	0.19	0.09	0.14
query26	0.31	0.14	0.14
query27	0.05	0.05	0.04
query28	13.83	1.04	0.45
query29	12.58	4.00	3.26
query30	0.26	0.09	0.06
query31	2.84	0.62	0.38
query32	3.23	0.55	0.46
query33	3.02	3.01	3.05
query34	16.72	5.17	4.57
query35	4.56	4.61	4.61
query36	0.67	0.50	0.47
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.04	0.03	0.02
query40	0.16	0.15	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 104.26 s
Total hot run time: 28.95 s

@wenzhenghu
Copy link
Contributor Author

wenzhenghu commented Oct 24, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 24, 2025

TPC-H: Total hot run time: 32399 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 30719547ae4e33d5ec553da6841a9980adfdec01, data reload: false

------ Round 1 ----------------------------------
q1	17579	5493	5443	5443
q2	2024	389	273	273
q3	12101	1217	737	737
q4	10209	863	440	440
q5	7950	2363	2096	2096
q6	186	167	133	133
q7	890	745	617	617
q8	9330	1437	1118	1118
q9	5241	4973	4907	4907
q10	6781	2272	1798	1798
q11	471	280	265	265
q12	325	355	211	211
q13	17785	3552	3000	3000
q14	232	222	213	213
q15	518	456	460	456
q16	433	424	364	364
q17	583	849	354	354
q18	6914	6337	6331	6331
q19	1225	954	522	522
q20	323	333	203	203
q21	2715	2132	1933	1933
q22	1049	1065	985	985
Total cold run time: 104864 ms
Total hot run time: 32399 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5559	5555	5497	5497
q2	243	340	241	241
q3	2278	2648	2332	2332
q4	1342	1757	1363	1363
q5	4428	4843	4901	4843
q6	184	159	128	128
q7	2078	1934	1863	1863
q8	2596	2805	2691	2691
q9	7275	7210	7266	7210
q10	3018	3303	2787	2787
q11	572	496	490	490
q12	676	759	626	626
q13	3359	3755	3118	3118
q14	289	297	281	281
q15	527	475	466	466
q16	447	483	439	439
q17	1217	1738	1243	1243
q18	7616	7533	7516	7516
q19	782	1050	1074	1050
q20	1997	2039	1912	1912
q21	5313	4955	4660	4660
q22	1129	1054	999	999
Total cold run time: 52925 ms
Total hot run time: 51755 ms

@doris-robot
Copy link

doris-robot commented Oct 24, 2025

TPC-DS: Total hot run time: 193177 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 30719547ae4e33d5ec553da6841a9980adfdec01, data reload: false

query1	951	410	414	410
query2	6262	2012	1981	1981
query3	8689	198	197	197
query4	33810	24036	23596	23596
query5	3713	599	448	448
query6	276	222	183	183
query7	4197	485	309	309
query8	305	236	233	233
query9	9315	2598	2587	2587
query10	506	325	273	273
query11	18070	15374	15466	15374
query12	156	110	113	110
query13	1561	546	436	436
query14	9018	6690	7203	6690
query15	262	196	179	179
query16	8526	617	508	508
query17	1564	771	627	627
query18	2151	408	324	324
query19	223	196	171	171
query20	128	131	121	121
query21	220	136	121	121
query22	4722	4645	4422	4422
query23	34664	34390	34681	34390
query24	7287	2685	2738	2685
query25	502	458	418	418
query26	749	278	169	169
query27	2097	464	376	376
query28	5635	2193	2157	2157
query29	662	581	446	446
query30	233	190	160	160
query31	1019	912	830	830
query32	88	59	58	58
query33	482	397	314	314
query34	755	848	516	516
query35	773	838	722	722
query36	1018	1072	959	959
query37	99	94	68	68
query38	4077	4083	3978	3978
query39	1528	1496	1468	1468
query40	200	117	107	107
query41	48	47	50	47
query42	125	107	103	103
query43	521	545	490	490
query44	1332	831	844	831
query45	183	184	172	172
query46	890	1073	673	673
query47	1976	2018	1950	1950
query48	415	422	356	356
query49	722	486	412	412
query50	671	684	429	429
query51	7359	7370	7295	7295
query52	105	108	97	97
query53	232	259	200	200
query54	555	572	488	488
query55	83	79	80	79
query56	279	272	263	263
query57	1313	1290	1260	1260
query58	239	219	221	219
query59	3216	3209	3183	3183
query60	314	303	282	282
query61	142	138	133	133
query62	815	765	724	724
query63	232	193	197	193
query64	3675	1094	746	746
query65	3379	3281	3317	3281
query66	860	463	323	323
query67	16411	15900	15593	15593
query68	6577	836	551	551
query69	517	315	274	274
query70	1220	1140	1181	1140
query71	366	291	261	261
query72	5778	3739	3867	3739
query73	629	741	361	361
query74	10503	9194	9032	9032
query75	3181	3150	2674	2674
query76	2990	1161	751	751
query77	543	369	291	291
query78	10492	10561	9597	9597
query79	3542	879	588	588
query80	678	513	438	438
query81	510	256	226	226
query82	578	118	90	90
query83	162	163	146	146
query84	238	106	85	85
query85	812	374	303	303
query86	393	334	271	271
query87	4275	4351	4298	4298
query88	5124	2435	2429	2429
query89	404	332	303	303
query90	1834	188	190	188
query91	138	140	108	108
query92	68	55	51	51
query93	2139	915	542	542
query94	686	415	283	283
query95	355	273	274	273
query96	487	614	298	298
query97	3181	3246	3137	3137
query98	220	202	207	202
query99	1571	1426	1304	1304
Total cold run time: 292776 ms
Total hot run time: 193177 ms

@doris-robot
Copy link

doris-robot commented Oct 24, 2025

ClickBench: Total hot run time: 28.66 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 30719547ae4e33d5ec553da6841a9980adfdec01, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.24	0.06	0.06
query4	1.63	0.11	0.10
query5	0.52	0.51	0.51
query6	1.12	0.73	0.73
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.60	0.49	0.51
query10	0.55	0.55	0.55
query11	0.15	0.11	0.10
query12	0.14	0.11	0.10
query13	0.61	0.62	0.60
query14	0.78	0.82	0.80
query15	0.85	0.84	0.83
query16	0.36	0.38	0.38
query17	1.04	1.05	1.02
query18	0.25	0.22	0.21
query19	1.97	1.73	1.84
query20	0.02	0.01	0.01
query21	15.39	0.94	0.56
query22	0.75	0.75	0.62
query23	15.16	1.47	0.53
query24	3.49	0.91	1.55
query25	0.11	0.23	0.08
query26	0.36	0.15	0.13
query27	0.06	0.04	0.04
query28	13.29	1.00	0.43
query29	12.58	3.86	3.26
query30	0.25	0.10	0.07
query31	2.81	0.61	0.38
query32	3.22	0.53	0.45
query33	3.03	2.95	3.08
query34	16.44	5.11	4.57
query35	4.64	4.55	4.58
query36	0.63	0.50	0.50
query37	0.08	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.12
query41	0.08	0.03	0.03
query42	0.04	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 103.67 s
Total hot run time: 28.66 s

@morrySnow morrySnow merged commit ca486c5 into apache:branch-3.1 Oct 28, 2025
20 of 21 checks passed
@morrySnow morrySnow mentioned this pull request Nov 13, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

dev/3.1.3-merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@wenzhenghu @hello-stephen @doris-robot @yiguolei @morrySnow