Skip to content

Conversation

@yujun777
Copy link
Contributor

@yujun777 yujun777 commented Aug 14, 2024

BUG: partition rebalancer migrates tablets back and forth: move from A to B, then B to A, then A to B, ... . The reason is the counting tablet num of backends is incorrect. It doesn't considering the pending and running sched tasks. After these tasks finished, the tablet num will change.

Fix: when calcuting the tablet num of backend, it should consider the in-progress moves which will change tablet num later.

@yujun777
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the doing label Aug 14, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 39903 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4f90bb9356fae776f7df7b95a89816d4fd9565ca, data reload: false

------ Round 1 ----------------------------------
q1	17611	4415	4267	4267
q2	2034	172	171	171
q3	10521	1175	1077	1077
q4	10136	712	699	699
q5	7766	2804	2794	2794
q6	227	141	141	141
q7	974	597	603	597
q8	9330	2053	1998	1998
q9	8663	6554	6548	6548
q10	7058	2245	2182	2182
q11	478	250	258	250
q12	389	219	218	218
q13	19010	2981	2970	2970
q14	281	235	228	228
q15	528	474	478	474
q16	518	411	377	377
q17	971	701	624	624
q18	8157	7428	7469	7428
q19	5129	1049	1069	1049
q20	665	340	328	328
q21	5501	4619	4486	4486
q22	1121	1003	997	997
Total cold run time: 117068 ms
Total hot run time: 39903 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4377	4309	4241	4241
q2	385	281	280	280
q3	2853	2687	2586	2586
q4	1949	1700	1694	1694
q5	5556	5690	5619	5619
q6	229	134	136	134
q7	2142	1759	1738	1738
q8	3249	3445	3389	3389
q9	8843	8725	8742	8725
q10	3466	3169	3203	3169
q11	607	501	500	500
q12	837	623	625	623
q13	16709	3106	3113	3106
q14	324	275	286	275
q15	518	481	491	481
q16	492	459	470	459
q17	1827	1534	1517	1517
q18	8068	7377	7369	7369
q19	1705	1487	1503	1487
q20	2072	1795	1806	1795
q21	9691	5100	5105	5100
q22	1095	1022	1023	1022
Total cold run time: 76994 ms
Total hot run time: 55309 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185809 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4f90bb9356fae776f7df7b95a89816d4fd9565ca, data reload: false

query1	923	387	373	373
query2	6442	2007	1895	1895
query3	6635	208	214	208
query4	33663	23257	23237	23237
query5	4230	520	485	485
query6	277	168	160	160
query7	4590	300	293	293
query8	259	206	202	202
query9	8425	2414	2382	2382
query10	434	289	287	287
query11	16255	15032	14931	14931
query12	148	96	100	96
query13	1631	360	349	349
query14	10131	7624	7904	7624
query15	219	167	165	165
query16	7706	477	504	477
query17	1521	543	551	543
query18	1954	291	284	284
query19	190	141	138	138
query20	111	102	101	101
query21	209	97	101	97
query22	4335	3973	4042	3973
query23	33814	33146	33223	33146
query24	11946	2952	2889	2889
query25	680	375	367	367
query26	1773	156	154	154
query27	2867	269	273	269
query28	7592	1991	1970	1970
query29	1081	439	470	439
query30	305	152	150	150
query31	959	753	741	741
query32	91	54	55	54
query33	736	275	279	275
query34	954	474	469	469
query35	827	731	710	710
query36	1107	948	944	944
query37	197	83	85	83
query38	4055	3810	3764	3764
query39	1450	1392	1391	1391
query40	286	120	113	113
query41	50	49	46	46
query42	117	98	97	97
query43	520	483	467	467
query44	1145	723	723	723
query45	193	161	162	161
query46	1104	738	745	738
query47	1806	1776	1751	1751
query48	372	293	292	292
query49	1175	439	414	414
query50	812	401	397	397
query51	6736	6745	6722	6722
query52	96	94	90	90
query53	257	184	186	184
query54	896	440	435	435
query55	77	73	76	73
query56	278	247	244	244
query57	1140	1073	1067	1067
query58	235	235	231	231
query59	2947	2911	2742	2742
query60	306	264	272	264
query61	100	98	100	98
query62	831	655	655	655
query63	206	188	187	187
query64	10657	2275	1740	1740
query65	3339	3195	3150	3150
query66	1374	331	332	331
query67	15120	14712	14736	14712
query68	4521	551	556	551
query69	392	281	266	266
query70	1107	1122	1153	1122
query71	401	285	294	285
query72	7199	2299	2089	2089
query73	757	319	326	319
query74	9306	8844	8818	8818
query75	3448	2647	2663	2647
query76	2877	1011	949	949
query77	459	306	326	306
query78	12194	9360	9065	9065
query79	2463	520	520	520
query80	1108	490	491	490
query81	576	228	228	228
query82	515	137	132	132
query83	269	145	146	145
query84	275	77	81	77
query85	710	307	276	276
query86	490	305	291	291
query87	4506	4149	4143	4143
query88	3831	2404	2443	2404
query89	397	288	286	286
query90	1995	205	194	194
query91	123	95	98	95
query92	62	54	49	49
query93	1440	545	544	544
query94	900	292	266	266
query95	351	261	257	257
query96	595	274	272	272
query97	3224	3063	3007	3007
query98	229	205	201	201
query99	1573	1276	1284	1276
Total cold run time: 301190 ms
Total hot run time: 185809 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.24 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4f90bb9356fae776f7df7b95a89816d4fd9565ca, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.05
query3	0.23	0.06	0.05
query4	1.67	0.08	0.07
query5	0.51	0.48	0.48
query6	1.13	0.73	0.74
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.54	0.53	0.54
query11	0.15	0.12	0.12
query12	0.14	0.13	0.12
query13	0.61	0.62	0.60
query14	0.78	0.81	0.79
query15	0.85	0.83	0.82
query16	0.36	0.38	0.36
query17	1.00	0.97	1.02
query18	0.23	0.22	0.23
query19	1.76	1.71	1.76
query20	0.01	0.01	0.02
query21	15.39	0.74	0.66
query22	4.59	7.56	1.37
query23	18.31	1.38	1.24
query24	2.07	0.24	0.22
query25	0.13	0.08	0.08
query26	0.30	0.22	0.21
query27	0.46	0.22	0.23
query28	13.29	1.02	1.00
query29	12.59	3.32	3.25
query30	0.24	0.06	0.06
query31	2.87	0.40	0.40
query32	3.27	0.49	0.48
query33	2.97	2.95	3.01
query34	16.88	4.43	4.38
query35	4.42	4.44	4.45
query36	0.66	0.49	0.51
query37	0.18	0.16	0.15
query38	0.15	0.15	0.15
query39	0.05	0.04	0.03
query40	0.15	0.13	0.13
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.88 s
Total hot run time: 30.24 s

@yujun777
Copy link
Contributor Author

run buildall

@yujun777
Copy link
Contributor Author

run buildall

1 similar comment
@yujun777
Copy link
Contributor Author

run buildall

Copy link
Contributor

@deardeng deardeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 37411 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8703450260877c131a38fa02f4e680c88214e4d4, data reload: false

------ Round 1 ----------------------------------
q1	17600	4380	4251	4251
q2	2017	182	183	182
q3	11904	1014	1144	1014
q4	10514	751	697	697
q5	7754	2794	2778	2778
q6	219	135	138	135
q7	950	582	591	582
q8	9511	2027	2043	2027
q9	8716	6494	6509	6494
q10	7038	2212	2156	2156
q11	468	241	243	241
q12	388	219	215	215
q13	17765	2988	2997	2988
q14	277	237	236	236
q15	535	488	489	488
q16	494	385	383	383
q17	957	703	745	703
q18	7453	6684	6898	6684
q19	7534	1137	993	993
q20	680	333	332	332
q21	3813	2846	3090	2846
q22	1063	1003	986	986
Total cold run time: 117650 ms
Total hot run time: 37411 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4572	4286	4295	4286
q2	377	266	271	266
q3	2862	2616	2592	2592
q4	1981	1618	1685	1618
q5	5632	5733	5568	5568
q6	224	136	140	136
q7	2127	1728	1706	1706
q8	3351	3483	3435	3435
q9	8804	8651	8817	8651
q10	3567	3302	3316	3302
q11	629	522	497	497
q12	782	641	604	604
q13	17032	3155	3210	3155
q14	308	286	295	286
q15	525	494	508	494
q16	495	451	448	448
q17	1868	1523	1535	1523
q18	8074	7848	7817	7817
q19	2822	1539	1524	1524
q20	2168	1892	1872	1872
q21	6972	5479	5534	5479
q22	1144	1002	1027	1002
Total cold run time: 76316 ms
Total hot run time: 56261 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189504 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8703450260877c131a38fa02f4e680c88214e4d4, data reload: false

query1	1257	883	843	843
query2	6429	1921	1841	1841
query3	10591	3921	3956	3921
query4	58261	26647	23201	23201
query5	5534	499	497	497
query6	468	176	155	155
query7	6231	320	291	291
query8	315	215	207	207
query9	8715	2448	2422	2422
query10	496	270	288	270
query11	17875	15013	15192	15013
query12	157	101	105	101
query13	1567	389	389	389
query14	11714	7811	7374	7374
query15	265	172	186	172
query16	7272	513	552	513
query17	1149	605	579	579
query18	1482	307	308	307
query19	314	147	148	147
query20	123	109	114	109
query21	221	105	108	105
query22	4561	4167	4609	4167
query23	34307	33306	33218	33218
query24	5748	2865	2836	2836
query25	552	405	407	405
query26	687	159	161	159
query27	1781	279	279	279
query28	3766	2079	2036	2036
query29	668	427	427	427
query30	239	151	154	151
query31	921	759	756	756
query32	79	60	57	57
query33	486	288	297	288
query34	863	467	463	463
query35	820	731	721	721
query36	1058	950	926	926
query37	144	84	87	84
query38	3879	3763	3827	3763
query39	1436	1377	1399	1377
query40	198	121	119	119
query41	49	45	48	45
query42	120	96	94	94
query43	505	457	458	457
query44	1096	747	733	733
query45	195	164	170	164
query46	1099	710	759	710
query47	1824	1718	1727	1718
query48	365	290	301	290
query49	771	428	439	428
query50	817	403	423	403
query51	6825	6628	6639	6628
query52	102	90	91	90
query53	262	183	185	183
query54	584	551	448	448
query55	73	76	75	75
query56	269	245	244	244
query57	1122	1062	1052	1052
query58	214	228	238	228
query59	2821	2603	2783	2603
query60	301	265	270	265
query61	95	93	96	93
query62	777	639	643	639
query63	219	184	182	182
query64	3366	1770	1736	1736
query65	3193	3123	3142	3123
query66	685	331	359	331
query67	15448	14878	14924	14878
query68	5065	542	551	542
query69	510	277	279	277
query70	1168	1139	1141	1139
query71	494	309	278	278
query72	6745	2292	1872	1872
query73	777	313	319	313
query74	9338	8793	8799	8793
query75	4497	2695	2758	2695
query76	3425	993	1020	993
query77	762	308	310	308
query78	9938	9096	8810	8810
query79	9637	551	552	551
query80	1046	514	503	503
query81	578	230	225	225
query82	629	141	137	137
query83	353	150	143	143
query84	267	73	74	73
query85	871	275	303	275
query86	347	269	307	269
query87	4375	4205	4070	4070
query88	4531	2287	2296	2287
query89	531	289	287	287
query90	2209	200	199	199
query91	124	98	98	98
query92	63	50	50	50
query93	6216	551	548	548
query94	877	296	289	289
query95	344	258	265	258
query96	616	268	267	267
query97	3165	3032	2993	2993
query98	228	204	193	193
query99	1523	1256	1240	1240
Total cold run time: 327075 ms
Total hot run time: 189504 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.99 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8703450260877c131a38fa02f4e680c88214e4d4, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.05
query3	0.23	0.05	0.04
query4	1.68	0.08	0.08
query5	0.52	0.50	0.50
query6	1.13	0.72	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.04
query9	0.54	0.48	0.48
query10	0.55	0.54	0.53
query11	0.16	0.11	0.11
query12	0.15	0.13	0.13
query13	0.62	0.61	0.60
query14	0.76	0.80	0.78
query15	0.85	0.82	0.82
query16	0.37	0.37	0.36
query17	1.00	1.05	0.95
query18	0.23	0.22	0.22
query19	1.87	1.81	1.82
query20	0.01	0.01	0.01
query21	15.39	0.74	0.65
query22	4.26	7.04	2.09
query23	18.28	1.37	1.24
query24	2.06	0.22	0.21
query25	0.15	0.09	0.08
query26	0.30	0.20	0.22
query27	0.46	0.24	0.23
query28	13.32	1.01	1.00
query29	12.62	3.37	3.35
query30	0.24	0.05	0.05
query31	2.88	0.39	0.39
query32	3.29	0.50	0.47
query33	2.96	2.95	2.97
query34	16.92	4.37	4.37
query35	4.46	4.43	4.40
query36	0.65	0.47	0.45
query37	0.19	0.16	0.15
query38	0.16	0.16	0.15
query39	0.04	0.04	0.03
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.85 s
Total hot run time: 30.99 s

@yujun777
Copy link
Contributor Author

run buildall

1 similar comment
@yujun777
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37578 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0cbc1719c141d9e01b1f410b3c215a6573d412b6, data reload: false

------ Round 1 ----------------------------------
q1	18184	4454	4375	4375
q2	2826	180	181	180
q3	12402	1160	1118	1118
q4	10620	751	796	751
q5	7780	2854	2724	2724
q6	224	142	137	137
q7	962	593	597	593
q8	9318	2038	2042	2038
q9	7068	6502	6485	6485
q10	7010	2144	2184	2144
q11	455	242	250	242
q12	388	224	221	221
q13	17768	3014	2971	2971
q14	283	233	230	230
q15	524	487	502	487
q16	480	393	379	379
q17	954	681	737	681
q18	7414	6822	6756	6756
q19	6784	1071	1071	1071
q20	701	321	341	321
q21	3834	2921	2684	2684
q22	1081	1009	990	990
Total cold run time: 117060 ms
Total hot run time: 37578 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4513	4232	4232	4232
q2	381	272	269	269
q3	2873	2612	2604	2604
q4	1903	1639	1628	1628
q5	5378	5351	5374	5351
q6	217	130	130	130
q7	2026	1631	1663	1631
q8	3132	3315	3302	3302
q9	8377	8333	8340	8333
q10	3377	3155	3152	3152
q11	598	499	500	499
q12	767	596	628	596
q13	17486	2996	2992	2992
q14	314	284	287	284
q15	518	471	490	471
q16	486	426	414	414
q17	1786	1518	1441	1441
q18	7631	7595	7342	7342
q19	1645	1467	1599	1467
q20	2011	1802	1787	1787
q21	5111	5138	4950	4950
q22	1107	1023	1012	1012
Total cold run time: 71637 ms
Total hot run time: 53887 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184812 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0cbc1719c141d9e01b1f410b3c215a6573d412b6, data reload: false

query1	905	389	380	380
query2	6444	2042	1941	1941
query3	6645	205	223	205
query4	34234	23172	23424	23172
query5	4225	505	507	505
query6	284	174	167	167
query7	4596	294	288	288
query8	242	211	199	199
query9	8478	2467	2423	2423
query10	436	279	267	267
query11	17322	14988	14991	14988
query12	153	99	100	99
query13	1652	375	355	355
query14	9533	6906	6870	6870
query15	217	162	165	162
query16	7911	491	483	483
query17	1563	556	526	526
query18	2051	285	278	278
query19	194	146	138	138
query20	111	107	106	106
query21	206	102	95	95
query22	4364	3992	3911	3911
query23	33847	33198	33155	33155
query24	11564	2801	2800	2800
query25	621	372	380	372
query26	1609	153	152	152
query27	2902	268	270	268
query28	7496	2054	2046	2046
query29	972	399	457	399
query30	307	150	149	149
query31	954	723	757	723
query32	95	54	55	54
query33	746	278	288	278
query34	947	461	468	461
query35	821	713	715	713
query36	1094	941	949	941
query37	142	79	83	79
query38	4059	3760	3795	3760
query39	1452	1404	1372	1372
query40	273	113	111	111
query41	52	44	45	44
query42	121	99	96	96
query43	506	492	496	492
query44	1232	733	731	731
query45	202	166	169	166
query46	1102	728	735	728
query47	1845	1729	1748	1729
query48	352	289	288	288
query49	1181	422	420	420
query50	804	405	409	405
query51	6772	6838	6614	6614
query52	115	89	91	89
query53	253	217	181	181
query54	839	446	450	446
query55	75	79	75	75
query56	268	243	242	242
query57	1163	1061	1082	1061
query58	243	221	280	221
query59	3146	2816	2814	2814
query60	282	256	265	256
query61	94	95	93	93
query62	835	652	646	646
query63	219	185	194	185
query64	6321	2295	1756	1756
query65	3244	3193	3161	3161
query66	1376	327	337	327
query67	15240	14811	14749	14749
query68	4639	545	539	539
query69	470	338	286	286
query70	1112	1149	1186	1149
query71	419	275	268	268
query72	6404	2232	2030	2030
query73	768	322	322	322
query74	9050	8892	8802	8802
query75	3430	2715	2749	2715
query76	2844	1005	956	956
query77	612	312	309	309
query78	9798	9053	8960	8960
query79	2456	550	526	526
query80	1970	503	488	488
query81	572	218	225	218
query82	869	134	143	134
query83	313	146	160	146
query84	280	78	80	78
query85	1934	276	267	267
query86	522	294	269	269
query87	4362	4215	4170	4170
query88	4315	2319	2375	2319
query89	390	286	286	286
query90	1847	197	190	190
query91	120	95	95	95
query92	65	49	50	49
query93	2535	532	526	526
query94	944	269	288	269
query95	349	251	252	251
query96	591	276	271	271
query97	3194	3035	3037	3035
query98	242	202	200	200
query99	1592	1283	1294	1283
Total cold run time: 298557 ms
Total hot run time: 184812 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.85 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0cbc1719c141d9e01b1f410b3c215a6573d412b6, data reload: false

query1	0.05	0.04	0.03
query2	0.09	0.04	0.04
query3	0.22	0.05	0.05
query4	1.69	0.08	0.09
query5	0.49	0.47	0.48
query6	1.13	0.72	0.72
query7	0.01	0.01	0.01
query8	0.05	0.04	0.05
query9	0.55	0.49	0.46
query10	0.55	0.53	0.54
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.61	0.62	0.58
query14	0.77	0.77	0.78
query15	0.86	0.82	0.82
query16	0.37	0.38	0.38
query17	1.00	1.02	0.98
query18	0.24	0.23	0.22
query19	1.80	1.70	1.70
query20	0.01	0.01	0.01
query21	15.42	0.76	0.66
query22	4.26	7.51	1.97
query23	18.30	1.39	1.30
query24	2.07	0.23	0.21
query25	0.16	0.09	0.08
query26	0.29	0.21	0.21
query27	0.46	0.23	0.22
query28	13.35	1.02	1.00
query29	12.60	3.40	3.38
query30	0.23	0.06	0.05
query31	2.91	0.40	0.39
query32	3.27	0.47	0.48
query33	2.95	2.96	3.00
query34	17.07	4.35	4.39
query35	4.42	4.49	4.43
query36	0.66	0.48	0.47
query37	0.20	0.16	0.15
query38	0.15	0.15	0.15
query39	0.05	0.04	0.04
query40	0.14	0.12	0.13
query41	0.09	0.04	0.05
query42	0.05	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.93 s
Total hot run time: 30.85 s

@yujun777
Copy link
Contributor Author

run cloud_p0

@yujun777
Copy link
Contributor Author

run cloud_p1

2 similar comments
@yujun777
Copy link
Contributor Author

run cloud_p1

@yujun777
Copy link
Contributor Author

run cloud_p1

@yujun777 yujun777 force-pushed the fix-partition-rebalancer-unstable branch from 0cbc171 to f863a2c Compare August 15, 2024 06:23
@yujun777
Copy link
Contributor Author

run buildall

@yujun777 yujun777 marked this pull request as draft August 15, 2024 06:31
@doris-robot
Copy link

TPC-H: Total hot run time: 37633 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f863a2c81395784b50674eea8288703031fc4c83, data reload: false

------ Round 1 ----------------------------------
q1	17909	4528	4438	4438
q2	2216	175	174	174
q3	10490	1180	1108	1108
q4	10144	767	680	680
q5	7752	2809	2740	2740
q6	224	137	136	136
q7	949	591	602	591
q8	9334	2081	2017	2017
q9	7259	6556	6535	6535
q10	7006	2120	2233	2120
q11	450	250	245	245
q12	398	227	224	224
q13	17765	2998	3108	2998
q14	299	264	238	238
q15	510	492	498	492
q16	483	397	399	397
q17	981	737	686	686
q18	7675	7006	6716	6716
q19	6812	1056	1011	1011
q20	697	318	344	318
q21	3905	2988	2760	2760
q22	1100	1036	1009	1009
Total cold run time: 114358 ms
Total hot run time: 37633 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4474	4221	4295	4221
q2	384	279	269	269
q3	2830	2628	2606	2606
q4	1897	1663	1596	1596
q5	5362	5390	5366	5366
q6	217	132	128	128
q7	2076	1661	1702	1661
q8	3171	3302	3333	3302
q9	8345	8367	8364	8364
q10	3412	3157	3156	3156
q11	586	527	517	517
q12	756	654	634	634
q13	17422	2995	2975	2975
q14	317	278	290	278
q15	524	482	477	477
q16	482	431	413	413
q17	1790	1509	1472	1472
q18	7828	7575	7443	7443
q19	1684	1621	1330	1330
q20	2013	1814	1799	1799
q21	5279	5043	5112	5043
q22	1100	1024	986	986
Total cold run time: 71949 ms
Total hot run time: 54036 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184864 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f863a2c81395784b50674eea8288703031fc4c83, data reload: false

query1	911	373	364	364
query2	6446	1950	1833	1833
query3	6653	208	217	208
query4	33192	23113	22980	22980
query5	4171	491	493	491
query6	263	161	154	154
query7	4603	295	286	286
query8	254	202	196	196
query9	8612	2434	2420	2420
query10	430	274	256	256
query11	17476	14928	15018	14928
query12	144	99	104	99
query13	1634	370	375	370
query14	9513	6647	6599	6599
query15	224	172	165	165
query16	7757	466	472	466
query17	1573	591	559	559
query18	1975	285	293	285
query19	199	147	144	144
query20	114	105	105	105
query21	209	103	105	103
query22	4438	4146	4094	4094
query23	33921	33366	33346	33346
query24	12201	2863	2898	2863
query25	687	392	394	392
query26	1827	160	155	155
query27	3040	274	279	274
query28	7869	2055	2062	2055
query29	1172	415	420	415
query30	306	149	155	149
query31	973	760	780	760
query32	100	57	56	56
query33	759	295	306	295
query34	943	458	461	458
query35	856	745	733	733
query36	1080	918	950	918
query37	290	84	79	79
query38	3905	3847	3883	3847
query39	1435	1356	1385	1356
query40	286	116	115	115
query41	49	46	45	45
query42	111	101	96	96
query43	491	464	471	464
query44	1165	735	726	726
query45	202	165	169	165
query46	1105	767	727	727
query47	1863	1773	1793	1773
query48	369	294	285	285
query49	1224	427	431	427
query50	808	400	401	400
query51	6815	6687	6662	6662
query52	98	89	94	89
query53	256	188	191	188
query54	944	448	448	448
query55	78	75	76	75
query56	275	258	265	258
query57	1156	1079	1111	1079
query58	259	222	231	222
query59	2864	2756	2760	2756
query60	304	270	279	270
query61	119	117	119	117
query62	835	649	650	649
query63	216	184	297	184
query64	6310	2266	1754	1754
query65	3178	3155	3140	3140
query66	1364	337	334	334
query67	15484	14903	14964	14903
query68	8599	545	560	545
query69	696	404	289	289
query70	1206	1065	1068	1065
query71	546	274	278	274
query72	7566	2225	2066	2066
query73	1508	318	315	315
query74	9271	8909	8768	8768
query75	5183	2617	2703	2617
query76	5094	1011	946	946
query77	790	312	299	299
query78	9669	9044	9318	9044
query79	8052	523	520	520
query80	1000	520	491	491
query81	590	222	225	222
query82	705	141	131	131
query83	324	148	147	147
query84	263	73	83	73
query85	1344	282	263	263
query86	393	307	288	288
query87	4363	4192	4327	4192
query88	4962	2292	2298	2292
query89	464	285	289	285
query90	2042	188	185	185
query91	121	92	96	92
query92	61	49	48	48
query93	5999	533	532	532
query94	984	300	278	278
query95	355	251	261	251
query96	608	274	267	267
query97	3243	3022	3098	3022
query98	225	195	206	195
query99	1564	1300	1255	1255
Total cold run time: 318173 ms
Total hot run time: 184864 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.29 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f863a2c81395784b50674eea8288703031fc4c83, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.08	0.07
query5	0.50	0.49	0.49
query6	1.12	0.74	0.72
query7	0.02	0.01	0.01
query8	0.06	0.05	0.05
query9	0.54	0.48	0.49
query10	0.55	0.54	0.53
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.62	0.59	0.59
query14	0.76	0.77	0.77
query15	0.85	0.82	0.83
query16	0.35	0.37	0.38
query17	0.97	0.99	1.05
query18	0.22	0.21	0.23
query19	1.90	1.80	1.75
query20	0.01	0.01	0.01
query21	15.40	0.75	0.65
query22	4.00	8.18	1.45
query23	18.29	1.37	1.26
query24	2.09	0.22	0.23
query25	0.15	0.08	0.09
query26	0.31	0.21	0.20
query27	0.46	0.22	0.23
query28	13.26	1.01	1.00
query29	12.67	3.30	3.33
query30	0.23	0.05	0.06
query31	2.88	0.40	0.39
query32	3.26	0.49	0.48
query33	2.90	2.99	2.96
query34	16.93	4.33	4.32
query35	4.40	4.48	4.41
query36	0.66	0.49	0.50
query37	0.19	0.16	0.15
query38	0.15	0.16	0.14
query39	0.05	0.04	0.04
query40	0.15	0.13	0.13
query41	0.08	0.04	0.05
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.4 s
Total hot run time: 30.29 s

@yujun777 yujun777 marked this pull request as ready for review August 19, 2024 08:33
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 19, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit bba7c2c into apache:master Aug 20, 2024
yujun777 added a commit to yujun777/doris that referenced this pull request Aug 20, 2024
… and forth (apache#39333)

BUG: partition rebalancer migrates tablets back and forth: move from A
to B, then B to A, then A to B, ... . The reason is the counting tablet
num of backends is incorrect. It doesn't considering the pending and
running sched tasks. After these tasks finished, the tablet num will
change.

Fix: when calcuting the tablet num of backend, it should consider the
in-progress moves which will change tablet num later.
yiguolei pushed a commit that referenced this pull request Aug 21, 2024
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
… and forth (#39333)

BUG: partition rebalancer migrates tablets back and forth: move from A
to B, then B to A, then A to B, ... . The reason is the counting tablet
num of backends is incorrect. It doesn't considering the pending and
running sched tasks. After these tasks finished, the tablet num will
change.

Fix: when calcuting the tablet num of backend, it should consider the
in-progress moves which will change tablet num later.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.3-merged doing reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants