Skip to content

Conversation

@deardeng
Copy link
Contributor

cherry pick from #52801

…ize is invalid (apache#52801)

…ize invalid

1. Fixed the problem that auto bucket will calculate wrong results when
partition size is inaccurate
- If `replica.size == 0`, filter out this replica. In the
tablet.getDataSize function, the size is calculated by taking the
average value of the replicas. When the size of a replica is 0, it will
have a great impact on the average value. Therefore, the replicas with
size=0 are filtered out.
- If the partition size equals 0, do not include it in the estimation of
the partition size.
- If all versions with data partitions have sizes equal to 0, then the
newly calculated bucket number for the partition will equal the bucket
number of the previous version with a size greater than 0. Since we do
not know the partition size of the data partitions (as stats thread have
not been collected yet), we assume that the new partition's size equals
the size of the previous version with a size greater than 0.
Consequently, the bucket number will naturally equal that of the
previous partition.

2. Added alarm log when the bucket num calculated by auto bucket exceeds
the threshold
@deardeng deardeng requested a review from morrySnow as a code owner July 15, 2025 02:53
@deardeng
Copy link
Contributor Author

run buildall

@Thearas
Copy link
Contributor

Thearas commented Jul 15, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@doris-robot
Copy link

TPC-H: Total hot run time: 39381 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a4b8ef31566b9621916263d05f2c3a776420bb35, data reload: false

------ Round 1 ----------------------------------
q1	17649	6855	6594	6594
q2	2052	179	196	179
q3	10517	1126	1156	1126
q4	10213	702	768	702
q5	7738	2874	2706	2706
q6	224	131	133	131
q7	975	625	609	609
q8	9365	1920	1966	1920
q9	6683	6410	6447	6410
q10	6998	2209	2274	2209
q11	459	260	250	250
q12	396	207	210	207
q13	17782	2958	2981	2958
q14	244	218	211	211
q15	496	474	493	474
q16	450	374	367	367
q17	972	536	529	529
q18	7322	6794	6662	6662
q19	1321	983	1062	983
q20	495	202	200	200
q21	3892	3064	2978	2978
q22	1104	1010	976	976
Total cold run time: 107347 ms
Total hot run time: 39381 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6559	6542	6547	6542
q2	328	231	229	229
q3	2927	2799	2968	2799
q4	2132	1830	1795	1795
q5	5648	5710	5715	5710
q6	209	127	129	127
q7	2181	1721	1727	1721
q8	3314	3474	3457	3457
q9	8943	8794	8846	8794
q10	3550	3538	3512	3512
q11	586	493	491	491
q12	792	575	617	575
q13	7035	3161	3145	3145
q14	312	289	268	268
q15	498	472	475	472
q16	475	428	456	428
q17	1831	1578	1564	1564
q18	8203	7839	7628	7628
q19	1653	1612	1499	1499
q20	2110	1891	1850	1850
q21	5347	4819	4738	4738
q22	1114	1002	999	999
Total cold run time: 65747 ms
Total hot run time: 58343 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190747 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a4b8ef31566b9621916263d05f2c3a776420bb35, data reload: false

query1	968	370	387	370
query2	6516	1948	1897	1897
query3	6711	215	223	215
query4	34214	23389	23962	23389
query5	4368	463	463	463
query6	275	173	197	173
query7	4632	332	316	316
query8	300	235	227	227
query9	9737	2592	2584	2584
query10	502	266	254	254
query11	18021	15323	15233	15233
query12	163	105	103	103
query13	1648	428	421	421
query14	8906	8001	7310	7310
query15	252	173	185	173
query16	7935	481	476	476
query17	1678	599	565	565
query18	1568	316	315	315
query19	375	161	160	160
query20	118	113	113	113
query21	203	107	106	106
query22	4478	4180	4242	4180
query23	35070	33832	33215	33215
query24	11289	2883	2888	2883
query25	686	412	421	412
query26	1474	169	169	169
query27	2792	349	346	346
query28	7510	2133	2132	2132
query29	957	460	443	443
query30	321	168	159	159
query31	1118	796	791	791
query32	101	66	70	66
query33	808	323	309	309
query34	963	499	511	499
query35	856	734	702	702
query36	1155	916	921	916
query37	136	76	73	73
query38	3940	3786	3827	3786
query39	1496	1410	1428	1410
query40	285	105	106	105
query41	54	51	53	51
query42	120	108	108	108
query43	535	485	464	464
query44	1440	812	802	802
query45	189	172	173	172
query46	1210	729	732	729
query47	1988	1876	1867	1867
query48	454	354	340	340
query49	1146	416	413	413
query50	894	415	417	415
query51	7405	7109	7209	7109
query52	108	97	96	96
query53	275	192	187	187
query54	1194	480	472	472
query55	83	80	82	80
query56	277	271	264	264
query57	1346	1170	1159	1159
query58	259	221	221	221
query59	3058	2952	2879	2879
query60	292	256	264	256
query61	118	108	114	108
query62	898	686	683	683
query63	216	200	191	191
query64	5294	660	633	633
query65	3309	3215	3164	3164
query66	1207	313	324	313
query67	15901	15729	15508	15508
query68	4847	576	584	576
query69	426	278	266	266
query70	1187	1073	1095	1073
query71	335	270	260	260
query72	6449	4179	4304	4179
query73	812	368	364	364
query74	10336	8942	8932	8932
query75	3480	2657	2632	2632
query76	2957	1072	1052	1052
query77	473	268	280	268
query78	10636	9605	9624	9605
query79	2031	606	599	599
query80	1137	434	421	421
query81	529	218	213	213
query82	939	89	86	86
query83	221	142	143	142
query84	244	79	76	76
query85	1310	307	301	301
query86	420	301	307	301
query87	4338	4186	4241	4186
query88	3868	2420	2377	2377
query89	422	293	290	290
query90	2020	187	184	184
query91	149	108	108	108
query92	75	53	52	52
query93	1598	571	561	561
query94	764	303	313	303
query95	360	260	255	255
query96	664	280	283	280
query97	3320	3177	3146	3146
query98	218	206	198	198
query99	1549	1329	1318	1318
Total cold run time: 301909 ms
Total hot run time: 190747 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.13 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a4b8ef31566b9621916263d05f2c3a776420bb35, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.03
query3	0.24	0.06	0.07
query4	1.62	0.10	0.11
query5	0.54	0.50	0.53
query6	1.14	0.74	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.56	0.50	0.50
query10	0.55	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	0.79	0.78	0.78
query15	0.84	0.84	0.81
query16	0.37	0.39	0.36
query17	1.00	0.98	1.05
query18	0.23	0.21	0.20
query19	1.95	1.88	1.85
query20	0.01	0.01	0.01
query21	15.41	0.59	0.57
query22	2.50	1.58	1.52
query23	16.81	1.13	0.88
query24	3.60	1.76	1.18
query25	0.13	0.12	0.18
query26	0.57	0.14	0.13
query27	0.05	0.03	0.04
query28	9.14	0.49	0.46
query29	12.55	3.23	3.23
query30	0.25	0.06	0.06
query31	2.87	0.37	0.39
query32	3.26	0.46	0.46
query33	2.96	3.00	3.01
query34	17.16	4.49	4.55
query35	4.54	4.53	4.57
query36	0.65	0.47	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.81 s
Total hot run time: 30.13 s

@morrySnow
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39465 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 15c8cd60705432eaeb25806e8d8cea0d320ebcbf, data reload: false

------ Round 1 ----------------------------------
q1	17588	7089	6605	6605
q2	2073	181	167	167
q3	10661	1132	1163	1132
q4	10525	736	719	719
q5	7738	2880	2740	2740
q6	213	133	134	133
q7	988	620	603	603
q8	9603	1930	2005	1930
q9	6947	6313	6395	6313
q10	7038	2220	2304	2220
q11	472	263	252	252
q12	398	213	211	211
q13	17774	2986	2985	2985
q14	239	206	207	206
q15	508	460	473	460
q16	481	385	383	383
q17	987	547	493	493
q18	7216	6680	6640	6640
q19	1335	1009	1011	1009
q20	489	203	205	203
q21	3960	3200	3090	3090
q22	1102	998	971	971
Total cold run time: 108335 ms
Total hot run time: 39465 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6656	6533	6569	6533
q2	327	234	226	226
q3	2912	2947	2905	2905
q4	2049	1869	1807	1807
q5	5687	5697	5700	5697
q6	209	127	129	127
q7	2202	1797	1841	1797
q8	3306	3493	3515	3493
q9	8730	8889	8831	8831
q10	3519	3487	3510	3487
q11	625	486	525	486
q12	812	604	610	604
q13	7121	3210	3119	3119
q14	307	274	265	265
q15	511	471	468	468
q16	504	435	431	431
q17	1841	1616	1601	1601
q18	8316	7703	7672	7672
q19	1675	1432	1704	1432
q20	2131	1829	1813	1813
q21	5276	5091	5010	5010
q22	1098	1042	1020	1020
Total cold run time: 65814 ms
Total hot run time: 58824 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196356 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 15c8cd60705432eaeb25806e8d8cea0d320ebcbf, data reload: false

query1	1274	936	894	894
query2	6303	1963	1869	1869
query3	10800	4334	4637	4334
query4	33130	23305	23620	23305
query5	3531	472	454	454
query6	278	183	190	183
query7	4016	324	322	322
query8	300	236	232	232
query9	9588	2609	2579	2579
query10	478	271	278	271
query11	17983	15161	15265	15161
query12	164	106	104	104
query13	1565	451	422	422
query14	8764	7787	6911	6911
query15	258	189	194	189
query16	8110	493	476	476
query17	1723	587	599	587
query18	2181	340	337	337
query19	391	172	173	172
query20	127	118	121	118
query21	215	116	108	108
query22	4789	4428	4462	4428
query23	35050	34367	33832	33832
query24	11303	2960	2888	2888
query25	668	445	427	427
query26	1201	174	178	174
query27	2480	362	363	362
query28	7326	2153	2173	2153
query29	889	479	475	475
query30	267	167	166	166
query31	1039	829	829	829
query32	98	55	59	55
query33	808	321	311	311
query34	974	520	526	520
query35	859	766	752	752
query36	1147	958	986	958
query37	143	73	69	69
query38	4278	4005	3981	3981
query39	1524	1455	1443	1443
query40	204	102	104	102
query41	48	45	54	45
query42	114	103	101	101
query43	523	495	495	495
query44	1289	849	806	806
query45	186	170	173	170
query46	1171	716	722	716
query47	2039	1870	1921	1870
query48	441	364	352	352
query49	976	409	396	396
query50	861	429	451	429
query51	7302	7216	7250	7216
query52	105	92	90	90
query53	259	184	182	182
query54	1250	481	478	478
query55	84	77	75	75
query56	277	277	257	257
query57	1339	1186	1209	1186
query58	234	220	221	220
query59	3177	3013	3017	3013
query60	290	266	261	261
query61	145	114	113	113
query62	845	681	696	681
query63	231	189	186	186
query64	3992	652	628	628
query65	3341	3335	3296	3296
query66	783	299	296	296
query67	16049	15495	15645	15495
query68	4598	590	616	590
query69	436	267	295	267
query70	1190	1128	1091	1091
query71	338	267	257	257
query72	6274	4083	3972	3972
query73	760	350	352	350
query74	10252	8939	9065	8939
query75	3420	2651	2690	2651
query76	2711	1021	1139	1021
query77	373	264	270	264
query78	10638	9529	9540	9529
query79	2394	611	602	602
query80	1198	435	420	420
query81	552	225	239	225
query82	842	89	95	89
query83	229	143	142	142
query84	228	79	75	75
query85	1390	315	292	292
query86	442	289	303	289
query87	4444	4310	4267	4267
query88	4304	2407	2372	2372
query89	421	296	292	292
query90	1988	189	190	189
query91	145	109	108	108
query92	66	51	53	51
query93	2280	566	560	560
query94	783	309	289	289
query95	356	266	265	265
query96	615	285	282	282
query97	3282	3123	3134	3123
query98	221	204	195	195
query99	1545	1292	1303	1292
Total cold run time: 302292 ms
Total hot run time: 196356 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.81 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 15c8cd60705432eaeb25806e8d8cea0d320ebcbf, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.11
query5	0.54	0.50	0.53
query6	1.14	0.72	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.55	0.50	0.51
query10	0.56	0.54	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	0.79	0.79	0.80
query15	0.86	0.83	0.82
query16	0.39	0.37	0.38
query17	1.02	1.01	1.06
query18	0.23	0.22	0.23
query19	1.82	1.85	1.79
query20	0.01	0.02	0.01
query21	15.42	0.61	0.57
query22	2.21	1.49	2.04
query23	17.08	0.88	0.81
query24	3.12	1.18	0.99
query25	0.30	0.27	0.12
query26	0.30	0.14	0.14
query27	0.04	0.04	0.05
query28	10.31	0.50	0.45
query29	12.57	3.26	3.30
query30	0.25	0.06	0.06
query31	2.87	0.39	0.39
query32	3.24	0.46	0.45
query33	3.03	3.01	3.00
query34	16.96	4.46	4.43
query35	4.53	4.50	4.48
query36	0.68	0.49	0.48
query37	0.08	0.06	0.05
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.18	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 104.2 s
Total hot run time: 29.81 s

@deardeng
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39838 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4a0eda836a2bdb13de38316940212a23505bf21e, data reload: false

------ Round 1 ----------------------------------
q1	17582	7110	6620	6620
q2	2311	212	178	178
q3	11966	1134	1257	1134
q4	10433	734	765	734
q5	7764	2918	2849	2849
q6	215	137	137	137
q7	998	625	610	610
q8	9590	1939	2023	1939
q9	7783	6385	6385	6385
q10	6995	2241	2270	2241
q11	451	256	266	256
q12	390	215	211	211
q13	17802	3027	2999	2999
q14	224	225	203	203
q15	496	464	458	458
q16	482	373	387	373
q17	979	553	570	553
q18	7402	6758	6756	6756
q19	1326	1060	962	962
q20	466	192	198	192
q21	3958	3241	3095	3095
q22	1088	967	953	953
Total cold run time: 110701 ms
Total hot run time: 39838 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6603	6571	6585	6571
q2	318	231	236	231
q3	3111	2883	2935	2883
q4	2107	1866	1796	1796
q5	5648	5740	5698	5698
q6	203	129	126	126
q7	2194	1821	1764	1764
q8	3347	3471	3485	3471
q9	8852	8874	8876	8874
q10	3594	3538	3522	3522
q11	605	497	489	489
q12	811	614	669	614
q13	6564	3161	3153	3153
q14	288	274	273	273
q15	503	463	466	463
q16	505	422	427	422
q17	1816	1587	1573	1573
q18	8092	7795	7622	7622
q19	1681	1488	1637	1488
q20	2150	1812	1784	1784
q21	5090	4818	4741	4741
q22	1091	1009	983	983
Total cold run time: 65173 ms
Total hot run time: 58541 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190625 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4a0eda836a2bdb13de38316940212a23505bf21e, data reload: false

query1	973	383	361	361
query2	6510	1910	1872	1872
query3	6708	215	228	215
query4	34066	23763	23767	23763
query5	4335	457	452	452
query6	262	171	178	171
query7	4612	319	308	308
query8	298	230	235	230
query9	9500	2597	2601	2597
query10	481	266	262	262
query11	18047	15529	15306	15306
query12	173	108	103	103
query13	1640	431	439	431
query14	9032	7290	7221	7221
query15	270	183	177	177
query16	8079	468	454	454
query17	1627	585	575	575
query18	2125	318	307	307
query19	259	159	158	158
query20	119	110	110	110
query21	211	107	105	105
query22	4396	4308	4282	4282
query23	34781	33244	33571	33244
query24	11549	2905	2845	2845
query25	715	411	408	408
query26	1867	170	178	170
query27	2963	351	344	344
query28	7803	2116	2109	2109
query29	1079	456	467	456
query30	325	156	155	155
query31	1030	774	785	774
query32	91	63	58	58
query33	787	304	293	293
query34	959	516	513	513
query35	838	693	708	693
query36	1066	948	911	911
query37	187	70	67	67
query38	3903	3829	3790	3790
query39	1509	1433	1421	1421
query40	285	102	100	100
query41	50	49	50	49
query42	125	111	106	106
query43	524	484	458	458
query44	1198	798	797	797
query45	193	175	177	175
query46	1141	709	737	709
query47	1891	1804	1853	1804
query48	426	342	340	340
query49	1250	403	383	383
query50	804	419	414	414
query51	7130	7206	7163	7163
query52	102	91	103	91
query53	262	185	184	184
query54	1197	473	477	473
query55	80	76	84	76
query56	270	253	258	253
query57	1316	1183	1124	1124
query58	235	214	215	214
query59	3082	2894	2885	2885
query60	287	258	258	258
query61	112	109	109	109
query62	831	671	666	666
query63	221	190	195	190
query64	5241	648	653	648
query65	3271	3178	3176	3176
query66	1287	327	302	302
query67	15780	15563	15487	15487
query68	4627	577	611	577
query69	425	267	264	264
query70	1147	1155	1038	1038
query71	332	254	267	254
query72	6353	4025	3895	3895
query73	764	349	358	349
query74	9897	8983	9153	8983
query75	3372	2678	2649	2649
query76	2975	1035	1063	1035
query77	391	286	292	286
query78	10296	9526	9644	9526
query79	1182	608	603	603
query80	769	435	431	431
query81	498	221	220	220
query82	1289	91	94	91
query83	244	151	151	151
query84	237	84	83	83
query85	1027	319	300	300
query86	319	306	296	296
query87	4319	4202	4229	4202
query88	3754	2437	2406	2406
query89	415	304	287	287
query90	1958	187	187	187
query91	144	112	110	110
query92	59	52	52	52
query93	1071	559	565	559
query94	721	290	274	274
query95	353	271	268	268
query96	609	289	282	282
query97	3315	3101	3123	3101
query98	219	206	197	197
query99	1501	1328	1324	1324
Total cold run time: 298348 ms
Total hot run time: 190625 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.44 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4a0eda836a2bdb13de38316940212a23505bf21e, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.62	0.11	0.11
query5	0.51	0.52	0.50
query6	1.12	0.72	0.74
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.56	0.51	0.51
query10	0.56	0.53	0.55
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.59	0.61
query14	0.77	0.79	0.80
query15	0.84	0.83	0.82
query16	0.40	0.37	0.38
query17	1.07	1.02	1.01
query18	0.23	0.21	0.22
query19	1.98	1.91	1.89
query20	0.01	0.01	0.01
query21	15.40	0.58	0.57
query22	2.73	2.50	2.48
query23	17.22	0.91	0.78
query24	3.41	0.72	1.05
query25	0.29	0.12	0.05
query26	0.47	0.13	0.13
query27	0.04	0.03	0.04
query28	10.44	0.54	0.46
query29	12.59	3.18	3.16
query30	0.25	0.06	0.07
query31	2.85	0.40	0.38
query32	3.22	0.46	0.46
query33	2.98	3.01	3.04
query34	17.27	4.47	4.46
query35	4.47	4.50	4.50
query36	0.66	0.47	0.47
query37	0.08	0.05	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.03
query40	0.15	0.12	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.7 s
Total hot run time: 30.44 s

@morrySnow morrySnow merged commit 9f16fba into apache:branch-3.1 Jul 16, 2025
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants