[fix](fe) Fix drop frontend cause `bdbje` and `fe memory` inconsistent state #35203

swjtu-zhanglei · 2024-05-22T07:52:26Z

dropFrontend use ReplicationGroupAdmin to remove bdbje node, in some corner case
ReplicationGroupAdmin.removeMember success but editLog.logRemoveFrontend fail
will cause inconsistent node info between fe memory and bdbje

doris-robot · 2024-05-22T07:52:31Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

swjtu-zhanglei · 2024-05-22T07:54:06Z

run buildall

doris-robot · 2024-05-22T08:31:59Z

TPC-H: Total hot run time: 40883 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b30a1652b04d3c8e24dabedd0b1bb8aa4a5b8e9e, data reload: false

------ Round 1 ----------------------------------
q1	17604	4374	4186	4186
q2	2020	194	197	194
q3	10452	1308	1158	1158
q4	10180	866	803	803
q5	7508	2720	2688	2688
q6	229	133	128	128
q7	966	620	609	609
q8	9229	2116	2082	2082
q9	10355	6692	6721	6692
q10	9176	3820	3894	3820
q11	463	248	255	248
q12	428	221	215	215
q13	17436	3182	3306	3182
q14	250	221	212	212
q15	524	461	468	461
q16	505	405	389	389
q17	988	676	724	676
q18	8337	8032	7808	7808
q19	7223	1581	1513	1513
q20	653	314	310	310
q21	5062	3225	4039	3225
q22	353	284	292	284
Total cold run time: 119941 ms
Total hot run time: 40883 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4643	4398	4466	4398
q2	388	275	290	275
q3	3166	2909	2772	2772
q4	1894	1622	1575	1575
q5	5486	5495	5489	5489
q6	221	127	128	127
q7	2142	1853	1854	1853
q8	3243	3374	3392	3374
q9	8558	8685	8623	8623
q10	3961	3740	3775	3740
q11	592	489	518	489
q12	786	638	626	626
q13	15910	3132	3140	3132
q14	317	266	264	264
q15	519	495	478	478
q16	499	436	456	436
q17	1805	1515	1490	1490
q18	7687	7611	7315	7315
q19	1646	1536	1549	1536
q20	1997	1794	1814	1794
q21	12577	4645	4591	4591
q22	600	502	474	474
Total cold run time: 78637 ms
Total hot run time: 54851 ms

doris-robot · 2024-05-22T08:42:38Z

TPC-DS: Total hot run time: 169083 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b30a1652b04d3c8e24dabedd0b1bb8aa4a5b8e9e, data reload: false

query1	926	379	358	358
query2	6453	2363	2233	2233
query3	6649	202	201	201
query4	19840	17359	17284	17284
query5	4103	428	421	421
query6	245	152	149	149
query7	4587	296	280	280
query8	245	184	185	184
query9	8546	2388	2363	2363
query10	451	287	257	257
query11	10635	10072	10121	10072
query12	133	87	87	87
query13	1645	368	379	368
query14	8564	7664	7618	7618
query15	221	170	179	170
query16	7666	263	255	255
query17	1686	514	505	505
query18	1748	267	270	267
query19	203	152	163	152
query20	95	83	83	83
query21	195	131	132	131
query22	4283	4005	3814	3814
query23	33548	32853	32972	32853
query24	8804	2759	2728	2728
query25	568	343	359	343
query26	705	151	153	151
query27	2237	318	327	318
query28	5861	2064	2043	2043
query29	859	601	590	590
query30	240	173	181	173
query31	960	750	739	739
query32	87	50	52	50
query33	632	288	289	288
query34	875	473	459	459
query35	718	599	582	582
query36	1057	898	891	891
query37	108	68	70	68
query38	2887	2800	2767	2767
query39	845	795	804	795
query40	195	128	130	128
query41	68	44	43	43
query42	103	95	99	95
query43	588	537	562	537
query44	1067	719	739	719
query45	177	161	166	161
query46	1062	690	721	690
query47	1855	1773	1776	1773
query48	370	293	287	287
query49	842	374	385	374
query50	755	387	374	374
query51	6913	6702	6817	6702
query52	109	86	92	86
query53	352	281	287	281
query54	777	432	425	425
query55	73	74	72	72
query56	249	240	243	240
query57	1114	1048	1004	1004
query58	230	216	248	216
query59	3317	3391	3187	3187
query60	271	245	252	245
query61	88	87	86	86
query62	603	455	448	448
query63	312	282	280	280
query64	8483	2273	1755	1755
query65	3192	3062	3104	3062
query66	790	335	325	325
query67	15050	14795	14935	14795
query68	4503	530	534	530
query69	444	272	276	272
query70	1150	1046	1117	1046
query71	370	262	267	262
query72	7358	2714	2529	2529
query73	702	315	313	313
query74	5906	5639	5642	5639
query75	3337	2636	2639	2636
query76	2268	975	953	953
query77	393	261	266	261
query78	10335	9684	9730	9684
query79	2054	508	508	508
query80	1215	454	449	449
query81	554	250	250	250
query82	666	101	93	93
query83	250	184	176	176
query84	244	159	82	82
query85	1840	272	255	255
query86	521	296	301	296
query87	3268	3120	3096	3096
query88	4329	2328	2306	2306
query89	473	391	396	391
query90	2058	188	185	185
query91	121	95	95	95
query92	60	49	47	47
query93	2765	502	491	491
query94	1251	188	183	183
query95	402	305	302	302
query96	590	269	274	269
query97	3220	2969	2980	2969
query98	246	226	212	212
query99	1197	823	854	823
Total cold run time: 263205 ms
Total hot run time: 169083 ms

doris-robot · 2024-05-22T08:48:09Z

ClickBench: Total hot run time: 30.58 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b30a1652b04d3c8e24dabedd0b1bb8aa4a5b8e9e, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.03	0.04
query3	0.24	0.05	0.04
query4	1.67	0.08	0.08
query5	0.50	0.49	0.51
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.54	0.49	0.48
query10	0.56	0.54	0.55
query11	0.16	0.11	0.11
query12	0.15	0.13	0.12
query13	0.60	0.59	0.61
query14	0.77	0.77	0.79
query15	0.83	0.82	0.82
query16	0.35	0.36	0.35
query17	0.98	0.97	0.98
query18	0.22	0.25	0.25
query19	1.87	1.72	1.68
query20	0.01	0.02	0.01
query21	15.73	0.66	0.65
query22	4.22	7.13	2.01
query23	18.28	1.31	1.24
query24	1.73	0.26	0.21
query25	0.15	0.08	0.08
query26	0.25	0.16	0.17
query27	0.08	0.08	0.07
query28	13.36	1.01	0.99
query29	13.33	3.34	3.30
query30	0.23	0.06	0.05
query31	2.86	0.38	0.38
query32	3.28	0.47	0.47
query33	2.84	2.93	2.89
query34	16.95	4.40	4.46
query35	4.48	4.49	4.55
query36	0.70	0.48	0.47
query37	0.18	0.15	0.16
query38	0.14	0.14	0.15
query39	0.05	0.03	0.04
query40	0.17	0.14	0.14
query41	0.09	0.05	0.04
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.96 s
Total hot run time: 30.58 s

swjtu-zhanglei · 2024-05-22T10:02:59Z

run buildall

doris-robot · 2024-05-22T10:21:10Z

TPC-H: Total hot run time: 39799 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 69362dbfeab969859ca8d6df1f60557de2dbd430, data reload: false

------ Round 1 ----------------------------------
q1	17665	4314	4214	4214
q2	2021	188	188	188
q3	10575	1230	1213	1213
q4	12371	772	849	772
q5	7527	2666	2636	2636
q6	217	135	134	134
q7	966	598	585	585
q8	9240	2083	2063	2063
q9	9568	6615	6580	6580
q10	8968	3698	3626	3626
q11	459	243	235	235
q12	418	213	211	211
q13	18757	2975	2972	2972
q14	247	219	222	219
q15	514	480	480	480
q16	508	370	384	370
q17	956	697	721	697
q18	7994	7382	7484	7382
q19	1903	1544	1512	1512
q20	639	294	304	294
q21	4930	3138	3860	3138
q22	346	278	284	278
Total cold run time: 116789 ms
Total hot run time: 39799 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4282	4253	4273	4253
q2	360	266	268	266
q3	2964	2797	2708	2708
q4	1854	1580	1577	1577
q5	5224	5260	5237	5237
q6	211	125	127	125
q7	2128	1722	1761	1722
q8	3168	3284	3292	3284
q9	8311	8266	8291	8266
q10	3818	3690	3682	3682
q11	572	483	497	483
q12	744	572	585	572
q13	17316	2968	2953	2953
q14	292	270	276	270
q15	514	482	482	482
q16	463	419	410	410
q17	1768	1484	1462	1462
q18	7743	7575	7451	7451
q19	1679	1586	1583	1583
q20	1987	1795	1766	1766
q21	4749	4748	4771	4748
q22	573	484	478	478
Total cold run time: 70720 ms
Total hot run time: 53778 ms

doris-robot · 2024-05-22T10:32:10Z

TPC-DS: Total hot run time: 172109 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 69362dbfeab969859ca8d6df1f60557de2dbd430, data reload: false

query1	928	372	366	366
query2	6960	2272	2367	2272
query3	6637	204	207	204
query4	19967	17628	17913	17628
query5	4136	414	407	407
query6	242	154	154	154
query7	4586	291	290	290
query8	238	182	179	179
query9	8702	2405	2392	2392
query10	446	271	257	257
query11	10698	10313	10036	10036
query12	129	92	87	87
query13	1630	348	353	348
query14	10180	7397	6973	6973
query15	219	167	171	167
query16	7869	256	252	252
query17	1866	525	522	522
query18	1918	271	276	271
query19	209	152	153	152
query20	90	82	82	82
query21	203	127	124	124
query22	4176	3913	3874	3874
query23	33587	33162	32995	32995
query24	11984	2880	2831	2831
query25	655	355	352	352
query26	1792	160	154	154
query27	2941	313	316	313
query28	7644	2029	2040	2029
query29	1079	618	587	587
query30	306	169	168	168
query31	999	747	757	747
query32	93	51	58	51
query33	771	263	257	257
query34	1041	465	461	461
query35	784	596	593	593
query36	1074	905	952	905
query37	271	72	67	67
query38	2939	2817	2768	2768
query39	836	803	789	789
query40	272	124	121	121
query41	45	43	44	43
query42	101	95	93	93
query43	604	561	539	539
query44	1166	719	748	719
query45	181	156	171	156
query46	1083	693	712	693
query47	1882	1775	1785	1775
query48	367	285	297	285
query49	1179	371	383	371
query50	767	376	379	376
query51	6912	6784	6821	6784
query52	102	96	90	90
query53	357	291	284	284
query54	1029	424	437	424
query55	76	72	71	71
query56	254	254	261	254
query57	1151	1078	1029	1029
query58	233	210	199	199
query59	3312	3244	3175	3175
query60	262	250	246	246
query61	91	87	85	85
query62	671	454	450	450
query63	305	280	281	280
query64	9798	2262	1734	1734
query65	3136	3094	3092	3092
query66	1358	336	322	322
query67	15316	14809	14918	14809
query68	4602	527	536	527
query69	437	262	259	259
query70	1103	1136	1047	1047
query71	402	265	256	256
query72	7590	5576	5518	5518
query73	717	316	313	313
query74	6036	5682	5624	5624
query75	3449	2675	2562	2562
query76	2797	986	912	912
query77	405	265	258	258
query78	10330	9715	9641	9641
query79	1994	509	511	509
query80	924	437	444	437
query81	540	241	243	241
query82	663	95	96	95
query83	243	170	163	163
query84	236	83	86	83
query85	1930	263	265	263
query86	510	310	326	310
query87	3340	3140	3133	3133
query88	4183	2319	2324	2319
query89	492	387	392	387
query90	2011	187	192	187
query91	136	154	97	97
query92	62	48	49	48
query93	2264	512	501	501
query94	1254	186	186	186
query95	404	305	306	305
query96	587	266	264	264
query97	3189	3027	3011	3011
query98	237	220	216	216
query99	1178	837	834	834
Total cold run time: 277080 ms
Total hot run time: 172109 ms

doris-robot · 2024-05-22T10:37:47Z

ClickBench: Total hot run time: 30.07 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 69362dbfeab969859ca8d6df1f60557de2dbd430, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.06	0.07
query5	0.50	0.49	0.50
query6	1.12	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.48
query10	0.54	0.54	0.54
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.60	0.58	0.60
query14	0.78	0.76	0.78
query15	0.82	0.81	0.80
query16	0.37	0.37	0.36
query17	1.02	1.01	0.98
query18	0.22	0.26	0.23
query19	1.86	1.78	1.81
query20	0.02	0.01	0.01
query21	15.74	0.69	0.65
query22	3.55	8.70	1.34
query23	18.28	1.40	1.32
query24	2.10	0.21	0.20
query25	0.15	0.09	0.08
query26	0.27	0.17	0.17
query27	0.08	0.08	0.07
query28	13.41	1.08	1.06
query29	12.56	3.33	3.28
query30	0.24	0.07	0.06
query31	2.87	0.39	0.38
query32	3.28	0.47	0.46
query33	2.88	2.92	2.84
query34	17.31	4.47	4.42
query35	4.56	4.52	4.51
query36	0.64	0.46	0.46
query37	0.18	0.15	0.15
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.16	0.14	0.14
query41	0.09	0.04	0.04
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.44 s
Total hot run time: 30.07 s

swjtu-zhanglei · 2024-05-23T02:32:37Z

run buildall

doris-robot · 2024-05-23T03:04:02Z

TPC-H: Total hot run time: 41840 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4994fbd447601609e93c8d7ff65240d8970a3173, data reload: false

------ Round 1 ----------------------------------
q1	17621	4337	4237	4237
q2	2027	188	188	188
q3	10475	1255	1263	1255
q4	10202	832	831	831
q5	7468	2674	2669	2669
q6	230	134	131	131
q7	965	621	593	593
q8	9219	2127	2086	2086
q9	9291	6699	6660	6660
q10	9284	3886	3867	3867
q11	447	245	246	245
q12	428	237	225	225
q13	18211	3150	3253	3150
q14	252	204	215	204
q15	512	471	462	462
q16	490	400	409	400
q17	956	670	689	670
q18	8468	7880	7926	7880
q19	5058	1571	1554	1554
q20	661	320	341	320
q21	5271	3922	4052	3922
q22	369	292	291	291
Total cold run time: 117905 ms
Total hot run time: 41840 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4592	4482	4456	4456
q2	390	260	254	254
q3	3152	2869	2808	2808
q4	1910	1715	1675	1675
q5	5486	5541	5521	5521
q6	220	126	129	126
q7	2215	1831	1809	1809
q8	3225	3426	3366	3366
q9	8630	8746	8902	8746
q10	4089	3963	4071	3963
q11	620	529	509	509
q12	852	648	673	648
q13	16403	3352	3259	3259
q14	315	287	269	269
q15	538	473	471	471
q16	486	432	437	432
q17	1807	1548	1488	1488
q18	8990	8252	7480	7480
q19	4495	1552	1508	1508
q20	2041	1803	1785	1785
q21	14208	4830	4689	4689
q22	568	492	488	488
Total cold run time: 85232 ms
Total hot run time: 55750 ms

doris-robot · 2024-05-23T03:14:53Z

TPC-DS: Total hot run time: 168818 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4994fbd447601609e93c8d7ff65240d8970a3173, data reload: false

query1	923	379	362	362
query2	6440	2397	2263	2263
query3	6642	209	212	209
query4	19226	17165	17347	17165
query5	4176	419	422	419
query6	237	154	164	154
query7	4580	294	283	283
query8	251	191	188	188
query9	8375	2384	2359	2359
query10	447	286	264	264
query11	10613	9956	9836	9836
query12	133	94	89	89
query13	1637	373	367	367
query14	9571	6601	7500	6601
query15	211	180	172	172
query16	7696	271	280	271
query17	1698	527	521	521
query18	1772	266	276	266
query19	186	153	163	153
query20	92	85	87	85
query21	191	130	131	130
query22	4318	3989	3999	3989
query23	33914	33039	33277	33039
query24	10992	2895	2851	2851
query25	608	356	356	356
query26	1137	161	158	158
query27	2855	321	330	321
query28	7591	2029	2061	2029
query29	888	627	607	607
query30	263	166	173	166
query31	936	741	741	741
query32	91	53	52	52
query33	750	273	272	272
query34	945	479	482	479
query35	712	644	604	604
query36	1072	924	883	883
query37	141	72	74	72
query38	2865	2779	2787	2779
query39	842	809	804	804
query40	193	124	123	123
query41	42	42	43	42
query42	101	98	97	97
query43	583	560	538	538
query44	1228	730	737	730
query45	183	168	158	158
query46	1068	758	721	721
query47	1853	1749	1771	1749
query48	367	289	300	289
query49	857	420	383	383
query50	765	396	410	396
query51	6948	6864	6862	6862
query52	99	90	86	86
query53	347	287	290	287
query54	833	427	422	422
query55	78	75	73	73
query56	252	240	244	240
query57	1099	1018	989	989
query58	247	200	212	200
query59	3447	3101	3160	3101
query60	275	271	248	248
query61	91	84	86	84
query62	621	453	449	449
query63	309	283	283	283
query64	8841	2249	1738	1738
query65	3168	3121	3152	3121
query66	779	338	325	325
query67	15165	14733	14807	14733
query68	5186	537	531	531
query69	494	262	315	262
query70	1189	1039	1075	1039
query71	490	271	274	271
query72	8092	5195	2744	2744
query73	741	323	324	323
query74	5959	5553	5584	5553
query75	3767	2624	2617	2617
query76	3752	992	1011	992
query77	582	282	261	261
query78	10233	9786	9806	9786
query79	2414	518	515	515
query80	1019	442	424	424
query81	536	239	246	239
query82	1296	101	98	98
query83	240	173	172	172
query84	244	87	81	81
query85	1564	272	286	272
query86	460	327	322	322
query87	3324	3097	3102	3097
query88	3737	2364	2346	2346
query89	483	400	388	388
query90	1999	193	196	193
query91	137	107	105	105
query92	63	51	49	49
query93	2177	519	494	494
query94	1294	199	192	192
query95	419	316	320	316
query96	603	267	272	267
query97	3193	2986	3020	2986
query98	257	224	223	223
query99	1222	834	851	834
Total cold run time: 273022 ms
Total hot run time: 168818 ms

doris-robot · 2024-05-23T03:20:22Z

ClickBench: Total hot run time: 30.57 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4994fbd447601609e93c8d7ff65240d8970a3173, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.70	0.08	0.08
query5	0.50	0.49	0.50
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.52	0.54	0.52
query11	0.14	0.11	0.11
query12	0.15	0.11	0.11
query13	0.60	0.59	0.60
query14	0.77	0.79	0.78
query15	0.82	0.82	0.82
query16	0.36	0.37	0.36
query17	0.95	1.00	0.94
query18	0.22	0.20	0.28
query19	1.76	1.74	1.67
query20	0.02	0.01	0.01
query21	15.73	0.65	0.63
query22	4.07	7.71	2.16
query23	18.36	1.37	1.21
query24	2.06	0.20	0.20
query25	0.14	0.08	0.09
query26	0.26	0.17	0.17
query27	0.08	0.07	0.08
query28	13.35	1.03	1.09
query29	13.29	3.24	3.24
query30	0.24	0.05	0.06
query31	2.88	0.38	0.39
query32	3.27	0.47	0.47
query33	2.82	2.85	2.92
query34	17.00	4.49	4.44
query35	4.58	4.49	4.65
query36	0.65	0.47	0.46
query37	0.17	0.15	0.16
query38	0.15	0.15	0.14
query39	0.05	0.04	0.03
query40	0.17	0.14	0.14
query41	0.08	0.05	0.04
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.09 s
Total hot run time: 30.57 s

…t state * dropFrontend use `ReplicationGroupAdmin` to remove bdbje node, in some corner case `ReplicationGroupAdmin.removeMember` success but `editLog.logRemoveFrontend` fail will cause inconsistent node info between `fe memory` and `bdbje`

swjtu-zhanglei · 2024-05-23T06:45:23Z

run buildall

doris-robot · 2024-05-23T07:10:45Z

TPC-H: Total hot run time: 39666 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 28eac70fa2dded64d0ee84d16361413e1d1a9608, data reload: false

------ Round 1 ----------------------------------
q1	17609	4357	4253	4253
q2	2010	190	195	190
q3	10458	1221	1133	1133
q4	10323	769	738	738
q5	7751	2680	2668	2668
q6	224	135	138	135
q7	975	619	603	603
q8	9634	2089	2080	2080
q9	9274	6608	6466	6466
q10	8957	3751	3731	3731
q11	453	235	229	229
q12	465	224	209	209
q13	17758	2941	2979	2941
q14	252	217	231	217
q15	502	462	469	462
q16	507	380	382	380
q17	943	628	700	628
q18	8175	7563	7538	7538
q19	2143	1524	1445	1445
q20	646	325	312	312
q21	4968	3034	3906	3034
q22	353	281	274	274
Total cold run time: 114380 ms
Total hot run time: 39666 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4320	4211	4185	4185
q2	383	258	255	255
q3	3001	2762	2748	2748
q4	1828	1556	1562	1556
q5	5230	5247	5265	5247
q6	210	121	127	121
q7	2144	1785	1679	1679
q8	3149	3293	3283	3283
q9	8272	8325	8311	8311
q10	3852	3647	3706	3647
q11	597	493	477	477
q12	732	607	576	576
q13	16369	2964	2964	2964
q14	293	249	255	249
q15	510	480	472	472
q16	460	420	414	414
q17	1765	1483	1471	1471
q18	7802	7647	7508	7508
q19	3594	1579	1533	1533
q20	2039	1792	1790	1790
q21	4827	4621	4765	4621
q22	567	483	482	482
Total cold run time: 71944 ms
Total hot run time: 53589 ms

doris-robot · 2024-05-23T07:21:42Z

TPC-DS: Total hot run time: 171461 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 28eac70fa2dded64d0ee84d16361413e1d1a9608, data reload: false

query1	923	374	369	369
query2	6467	2569	2360	2360
query3	6655	208	205	205
query4	19535	17606	17258	17258
query5	4220	418	418	418
query6	257	170	156	156
query7	4591	300	285	285
query8	257	189	186	186
query9	8492	2377	2347	2347
query10	459	270	277	270
query11	10613	10110	10151	10110
query12	139	93	88	88
query13	1650	370	361	361
query14	9300	6298	6842	6298
query15	216	172	172	172
query16	7723	272	269	269
query17	1390	534	518	518
query18	1946	273	269	269
query19	197	155	150	150
query20	90	85	83	83
query21	192	128	131	128
query22	4181	4267	3906	3906
query23	33762	32874	32909	32874
query24	12396	2752	2961	2752
query25	679	350	364	350
query26	1854	157	156	156
query27	3070	318	319	318
query28	7755	2035	2054	2035
query29	1188	625	586	586
query30	273	172	172	172
query31	980	761	776	761
query32	90	54	52	52
query33	760	270	263	263
query34	1036	478	459	459
query35	720	592	610	592
query36	1068	921	941	921
query37	276	69	70	69
query38	2939	2788	2777	2777
query39	868	785	814	785
query40	268	125	124	124
query41	46	45	43	43
query42	106	96	95	95
query43	589	523	543	523
query44	1302	761	737	737
query45	183	163	161	161
query46	1069	728	692	692
query47	1876	1796	1759	1759
query48	379	285	297	285
query49	1185	383	387	383
query50	769	376	381	376
query51	6844	6690	6674	6674
query52	101	90	90	90
query53	353	289	278	278
query54	938	434	414	414
query55	76	76	76	76
query56	265	238	246	238
query57	1169	1040	1031	1031
query58	232	209	213	209
query59	3549	3290	3334	3290
query60	275	256	273	256
query61	89	88	87	87
query62	669	450	454	450
query63	309	281	281	281
query64	9788	2211	1820	1820
query65	3199	3097	3146	3097
query66	1374	338	321	321
query67	15279	14960	14907	14907
query68	4539	530	544	530
query69	433	263	267	263
query70	1147	1149	1086	1086
query71	409	275	261	261
query72	7617	5408	5574	5408
query73	720	322	323	322
query74	5982	5593	5575	5575
query75	3478	2597	2615	2597
query76	2745	1008	1011	1008
query77	390	281	265	265
query78	10462	9750	9842	9750
query79	2187	518	515	515
query80	1074	434	431	431
query81	519	242	244	242
query82	642	92	100	92
query83	243	169	166	166
query84	236	85	91	85
query85	1874	272	265	265
query86	506	309	314	309
query87	3282	3171	3107	3107
query88	4180	2328	2327	2327
query89	475	405	389	389
query90	1998	184	185	184
query91	127	96	98	96
query92	64	49	52	49
query93	2381	504	492	492
query94	1264	186	183	183
query95	399	314	308	308
query96	588	269	261	261
query97	3177	2997	3035	2997
query98	234	215	219	215
query99	1205	858	860	858
Total cold run time: 276065 ms
Total hot run time: 171461 ms

doris-robot · 2024-05-23T07:27:13Z

ClickBench: Total hot run time: 29.81 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 28eac70fa2dded64d0ee84d16361413e1d1a9608, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.03	0.03
query3	0.22	0.05	0.05
query4	1.69	0.06	0.07
query5	0.47	0.50	0.49
query6	1.12	0.72	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.49	0.50
query10	0.55	0.54	0.54
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.60	0.60	0.60
query14	0.76	0.77	0.77
query15	0.83	0.80	0.79
query16	0.35	0.37	0.34
query17	0.96	0.96	1.04
query18	0.22	0.24	0.25
query19	1.85	1.75	1.67
query20	0.01	0.00	0.01
query21	15.73	0.67	0.66
query22	4.69	7.34	1.48
query23	18.27	1.33	1.22
query24	1.44	0.36	0.22
query25	0.15	0.08	0.08
query26	0.25	0.16	0.17
query27	0.08	0.08	0.08
query28	13.37	1.01	0.99
query29	13.20	3.28	3.22
query30	0.24	0.06	0.06
query31	2.86	0.39	0.38
query32	3.32	0.47	0.46
query33	2.88	2.86	2.93
query34	17.16	4.36	4.37
query35	4.49	4.54	4.48
query36	0.67	0.47	0.50
query37	0.18	0.15	0.16
query38	0.15	0.16	0.14
query39	0.04	0.04	0.03
query40	0.17	0.14	0.16
query41	0.08	0.04	0.04
query42	0.05	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 110.17 s
Total hot run time: 29.81 s

github-actions · 2024-05-24T06:39:10Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

github-actions · 2024-05-28T08:42:43Z

PR approved by at least one committer and no changes requested.

An exception introduced by #35203 cause the frontend start failed In this cases: 1. all frontends are down and recover frontend in different host by the original metadata 2. or the only frontend's ip changed, and start with `-r` option to recover from original metadata

…t state (#35203) * dropFrontend use `ReplicationGroupAdmin` to remove bdbje node, in some corner case `ReplicationGroupAdmin.removeMember` success but `editLog.logRemoveFrontend` fail will cause inconsistent node info between `fe memory` and `bdbje`

An exception introduced by #35203 cause the frontend start failed In this cases: 1. all frontends are down and recover frontend in different host by the original metadata 2. or the only frontend's ip changed, and start with `-r` option to recover from original metadata

``` com.sleepycat.je.rep.MasterStateException: (JE 18.3.12) Cannot remove an active master at com.sleepycat.je.rep.util.ReplicationGroupAdmin.checkMember(ReplicationGroupAdmin.java:591) at com.sleepycat.je.rep.util.ReplicationGroupAdmin.removeMember(ReplicationGroupAdmin.java:315) at org.apache.doris.ha.BDBHA.removeDroppedMember(BDBHA.java:262) at org.apache.doris.catalog.Env.removeDroppedFrontends(Env.java:3077) at org.apache.doris.catalog.Env.transferToMaster(Env.java:1520) at org.apache.doris.catalog.Env.access$1300(Env.java:338) at org.apache.doris.catalog.Env$5.runOneCycle(Env.java:2792) at org.apache.doris.common.util.Daemon.run(Daemon.java:116) ``` * the bug is introduced by apache#35203

``` com.sleepycat.je.rep.MasterStateException: (JE 18.3.12) Cannot remove an active master at com.sleepycat.je.rep.util.ReplicationGroupAdmin.checkMember(ReplicationGroupAdmin.java:591) at com.sleepycat.je.rep.util.ReplicationGroupAdmin.removeMember(ReplicationGroupAdmin.java:315) at org.apache.doris.ha.BDBHA.removeDroppedMember(BDBHA.java:262) at org.apache.doris.catalog.Env.removeDroppedFrontends(Env.java:3077) at org.apache.doris.catalog.Env.transferToMaster(Env.java:1520) at org.apache.doris.catalog.Env.access$1300(Env.java:338) at org.apache.doris.catalog.Env$5.runOneCycle(Env.java:2792) at org.apache.doris.common.util.Daemon.run(Daemon.java:116) ``` * the bug is introduced by #35203

swjtu-zhanglei force-pushed the master-fix4 branch from b30a165 to 69362db Compare May 22, 2024 10:02

swjtu-zhanglei force-pushed the master-fix4 branch 2 times, most recently from c70f678 to 4994fbd Compare May 23, 2024 02:31

swjtu-zhanglei marked this pull request as ready for review May 23, 2024 02:32

swjtu-zhanglei force-pushed the master-fix4 branch from 4994fbd to 28eac70 Compare May 23, 2024 06:44

gavinchou approved these changes May 24, 2024

View reviewed changes

gavinchou added the dev/2.0.x label May 24, 2024

github-actions bot added the reviewed label May 24, 2024

dataroaring approved these changes May 28, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 28, 2024

dataroaring merged commit 6aa57e5 into apache:master May 28, 2024

dataroaring added the dev/2.1.x label May 28, 2024

924060929 mentioned this pull request May 29, 2024

[fix](fe) fix start fe failed when recover fe in different host #35612

Merged

morrySnow added not-merge/2.0 do not merge into 2.0 branch not-merge/2.1 and removed dev/2.0.x dev/2.1.x labels Jun 3, 2024

swjtu-zhanglei mentioned this pull request Jul 17, 2024

[fix](fe) Fix removeDroppedMember use incorrect nodeName #37987

Merged

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

[fix](fe) Fix drop frontend cause bdbje and fe memory inconsistent state #35203

[fix](fe) Fix drop frontend cause bdbje and fe memory inconsistent state #35203

Uh oh!

Conversation

swjtu-zhanglei commented May 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

swjtu-zhanglei commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

swjtu-zhanglei commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

swjtu-zhanglei commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

swjtu-zhanglei commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

doris-robot commented May 23, 2024

Uh oh!

github-actions bot commented May 24, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[fix](fe) Fix drop frontend cause `bdbje` and `fe memory` inconsistent state #35203

[fix](fe) Fix drop frontend cause `bdbje` and `fe memory` inconsistent state #35203

swjtu-zhanglei commented May 22, 2024 •

edited

Loading