Skip to content

Conversation

@eldenmoon
Copy link
Member

@eldenmoon eldenmoon commented Jul 10, 2025

What problem does this PR solve?

improve performance of caculate stats
Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@eldenmoon eldenmoon requested a review from morrySnow as a code owner July 10, 2025 17:20
@Thearas
Copy link
Contributor

Thearas commented Jul 10, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@eldenmoon
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39954 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3b925ae481291527f19a5f41c5b521dedb41099a, data reload: false

------ Round 1 ----------------------------------
q1	17649	6737	6656	6656
q2	2057	202	207	202
q3	10481	1132	1188	1132
q4	10213	792	764	764
q5	7777	2892	2862	2862
q6	216	134	136	134
q7	1003	622	596	596
q8	9577	1988	2017	1988
q9	8126	6382	6431	6382
q10	7043	2217	2275	2217
q11	457	254	258	254
q12	394	215	211	211
q13	17797	2987	2993	2987
q14	227	211	204	204
q15	498	468	471	468
q16	469	376	365	365
q17	1004	583	552	552
q18	7337	6648	6691	6648
q19	1322	1039	1036	1036
q20	483	202	203	202
q21	3900	3129	3164	3129
q22	1104	965	982	965
Total cold run time: 109134 ms
Total hot run time: 39954 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6590	6572	6557	6557
q2	324	226	236	226
q3	2953	2958	2898	2898
q4	2081	1883	1805	1805
q5	5690	5696	5767	5696
q6	206	131	131	131
q7	2226	1811	1805	1805
q8	3323	3535	3530	3530
q9	8832	8870	8909	8870
q10	3556	3551	3517	3517
q11	589	479	503	479
q12	815	599	598	598
q13	6493	3139	3148	3139
q14	317	267	274	267
q15	503	468	474	468
q16	494	432	425	425
q17	1853	1645	1611	1611
q18	8294	7847	7876	7847
q19	1692	1570	1533	1533
q20	2028	1817	1857	1817
q21	5172	5035	5033	5033
q22	1120	1063	1029	1029
Total cold run time: 65151 ms
Total hot run time: 59281 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196159 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3b925ae481291527f19a5f41c5b521dedb41099a, data reload: false

query1	1319	894	918	894
query2	6236	1993	1867	1867
query3	10839	4360	4563	4360
query4	33308	23568	23865	23568
query5	4199	465	446	446
query6	276	185	180	180
query7	4001	318	336	318
query8	291	229	216	216
query9	9559	2572	2561	2561
query10	477	269	262	262
query11	17839	15160	15553	15160
query12	160	104	105	104
query13	1550	438	420	420
query14	9650	6665	6977	6665
query15	228	191	196	191
query16	7810	519	519	519
query17	1680	593	624	593
query18	2067	314	324	314
query19	256	175	172	172
query20	130	127	126	126
query21	204	112	113	112
query22	4794	4389	4438	4389
query23	34678	34432	34135	34135
query24	11575	2926	2904	2904
query25	686	429	430	429
query26	1638	174	170	170
query27	2994	354	354	354
query28	7568	2182	2156	2156
query29	1032	468	479	468
query30	274	162	162	162
query31	1029	863	886	863
query32	100	63	60	60
query33	800	335	288	288
query34	1000	503	510	503
query35	864	754	723	723
query36	1115	935	908	908
query37	122	66	68	66
query38	4075	3917	3915	3915
query39	1502	1478	1486	1478
query40	254	100	115	100
query41	52	48	48	48
query42	125	100	101	100
query43	525	492	484	484
query44	1256	844	818	818
query45	183	168	174	168
query46	1174	736	728	728
query47	2062	1907	1949	1907
query48	439	339	348	339
query49	1183	393	391	391
query50	842	428	417	417
query51	7417	7248	7144	7144
query52	103	96	94	94
query53	262	182	181	181
query54	1339	489	487	487
query55	76	79	75	75
query56	282	264	268	264
query57	1368	1204	1202	1202
query58	249	218	219	218
query59	3208	3017	3017	3017
query60	282	259	262	259
query61	137	103	107	103
query62	862	671	693	671
query63	223	186	200	186
query64	4912	660	662	660
query65	3334	3270	3273	3270
query66	1366	297	304	297
query67	16370	15787	15546	15546
query68	5209	582	571	571
query69	425	265	266	265
query70	1178	1117	1173	1117
query71	337	265	258	258
query72	6258	4002	3720	3720
query73	745	347	354	347
query74	10569	8909	9091	8909
query75	3364	2672	2690	2672
query76	2986	1183	983	983
query77	391	263	267	263
query78	10418	9530	9515	9515
query79	1324	592	586	586
query80	1033	428	420	420
query81	544	226	223	223
query82	1000	89	91	89
query83	243	145	149	145
query84	242	80	84	80
query85	1335	341	293	293
query86	385	302	289	289
query87	4292	4239	4237	4237
query88	3676	2396	2388	2388
query89	416	296	294	294
query90	1998	184	181	181
query91	144	111	106	106
query92	58	49	53	49
query93	1447	554	564	554
query94	892	308	260	260
query95	354	261	255	255
query96	607	283	292	283
query97	3264	3106	3141	3106
query98	212	204	198	198
query99	1531	1310	1313	1310
Total cold run time: 304687 ms
Total hot run time: 196159 ms

@eldenmoon eldenmoon changed the title variant stats caculator [Improvement](variant) add stats caculator Jul 10, 2025
@eldenmoon
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39855 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 45f8e363289f297ca3f78c61576d225f4400c7ec, data reload: false

------ Round 1 ----------------------------------
q1	17610	6843	6636	6636
q2	2073	188	204	188
q3	10565	1145	1140	1140
q4	10240	713	676	676
q5	7708	2984	2887	2887
q6	213	137	136	136
q7	1006	635	611	611
q8	9361	2004	2022	2004
q9	6597	6405	6466	6405
q10	7024	2289	2308	2289
q11	466	257	267	257
q12	391	212	207	207
q13	17809	3021	2982	2982
q14	241	217	213	213
q15	521	463	463	463
q16	486	374	383	374
q17	995	638	527	527
q18	7357	6647	6792	6647
q19	1320	1042	1012	1012
q20	488	203	193	193
q21	4276	3218	3036	3036
q22	1071	1011	972	972
Total cold run time: 107818 ms
Total hot run time: 39855 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6620	6592	6594	6592
q2	332	237	243	237
q3	2970	2930	2909	2909
q4	2054	1797	1764	1764
q5	5671	5716	5736	5716
q6	215	128	130	128
q7	2223	1778	1834	1778
q8	3394	3484	3550	3484
q9	8779	8974	8971	8971
q10	3561	3506	3487	3487
q11	593	493	510	493
q12	808	612	589	589
q13	8210	3227	3141	3141
q14	296	284	272	272
q15	514	467	459	459
q16	489	432	436	432
q17	1832	1642	1599	1599
q18	8201	7753	7718	7718
q19	1685	1590	1603	1590
q20	2055	1831	1860	1831
q21	5169	4939	4965	4939
q22	1117	1071	1035	1035
Total cold run time: 66788 ms
Total hot run time: 59164 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196305 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 45f8e363289f297ca3f78c61576d225f4400c7ec, data reload: false

query1	1283	958	912	912
query2	6319	1958	1879	1879
query3	10813	4304	4379	4304
query4	32931	23590	23367	23367
query5	3536	488	452	452
query6	275	176	176	176
query7	3988	308	312	308
query8	293	239	233	233
query9	9475	2619	2602	2602
query10	485	269	258	258
query11	18497	15202	15109	15109
query12	180	104	103	103
query13	1561	437	437	437
query14	9500	7099	6644	6644
query15	258	188	203	188
query16	7964	479	523	479
query17	1640	620	602	602
query18	2189	343	321	321
query19	249	161	172	161
query20	132	117	117	117
query21	202	116	110	110
query22	4677	4434	4440	4434
query23	35895	34350	34050	34050
query24	12338	2916	2969	2916
query25	615	427	441	427
query26	1377	171	184	171
query27	2872	362	348	348
query28	7655	2187	2156	2156
query29	690	450	455	450
query30	261	162	162	162
query31	1025	798	839	798
query32	97	59	58	58
query33	757	307	295	295
query34	1030	500	529	500
query35	875	732	708	708
query36	1063	939	944	939
query37	225	70	70	70
query38	4143	3950	3929	3929
query39	1517	1450	1480	1450
query40	255	101	101	101
query41	47	47	47	47
query42	121	106	107	106
query43	520	480	493	480
query44	1268	844	842	842
query45	193	172	173	172
query46	1158	742	714	714
query47	1997	1939	1903	1903
query48	434	346	361	346
query49	1021	404	419	404
query50	866	442	429	429
query51	7402	7347	7292	7292
query52	109	96	95	95
query53	257	183	184	183
query54	1407	481	500	481
query55	83	77	79	77
query56	272	251	254	251
query57	1337	1218	1218	1218
query58	232	217	223	217
query59	3291	2961	3057	2961
query60	291	273	264	264
query61	116	115	107	107
query62	878	707	691	691
query63	215	192	206	192
query64	4978	685	632	632
query65	3433	3346	3318	3318
query66	1298	305	295	295
query67	15796	15349	15464	15349
query68	5699	570	584	570
query69	425	266	275	266
query70	1200	1071	1087	1071
query71	333	262	266	262
query72	5950	4028	4012	4012
query73	744	360	363	360
query74	10579	9013	9278	9013
query75	3382	2670	2679	2670
query76	3196	1134	1032	1032
query77	402	275	292	275
query78	10446	9554	9495	9495
query79	1407	588	594	588
query80	1041	432	423	423
query81	537	225	229	225
query82	955	93	96	93
query83	244	145	147	145
query84	237	87	77	77
query85	1339	383	393	383
query86	397	315	299	299
query87	4373	4213	4239	4213
query88	3584	2427	2407	2407
query89	420	296	299	296
query90	1919	191	196	191
query91	144	110	105	105
query92	61	53	52	52
query93	1444	547	560	547
query94	896	303	303	303
query95	382	265	262	262
query96	613	281	288	281
query97	3282	3126	3245	3126
query98	227	209	200	200
query99	1518	1308	1273	1273
Total cold run time: 305533 ms
Total hot run time: 196305 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.81 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 45f8e363289f297ca3f78c61576d225f4400c7ec, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.07
query4	1.63	0.10	0.10
query5	0.53	0.51	0.54
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.55	0.52	0.51
query10	0.55	0.55	0.56
query11	0.14	0.10	0.11
query12	0.14	0.11	0.10
query13	0.60	0.60	0.59
query14	0.78	0.82	0.77
query15	0.83	0.84	0.82
query16	0.39	0.39	0.39
query17	1.11	1.05	1.05
query18	0.23	0.20	0.21
query19	1.95	1.87	1.84
query20	0.01	0.01	0.01
query21	15.42	0.60	0.58
query22	2.18	1.51	1.80
query23	17.23	0.94	0.76
query24	2.93	1.52	0.99
query25	0.24	0.05	0.22
query26	0.46	0.14	0.14
query27	0.04	0.05	0.04
query28	10.18	0.51	0.44
query29	12.55	3.20	3.21
query30	0.24	0.06	0.05
query31	2.89	0.40	0.37
query32	3.22	0.46	0.46
query33	3.03	3.00	3.03
query34	17.05	4.51	4.51
query35	4.52	4.51	4.49
query36	0.68	0.48	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.31 s
Total hot run time: 29.81 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 96.49% (55/57) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.21% (12506/27659)
Line Coverage 36.13% (111088/307500)
Region Coverage 35.25% (57463/163034)
Branch Coverage 32.35% (31204/96444)

@morrySnow morrySnow merged commit 00f8f74 into apache:branch-3.1 Jul 11, 2025
21 of 23 checks passed
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Jul 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants