[feat](storage vault) Check storage vault connectivity for BE when starting #51175

swjtu-zhanglei · 2025-05-23T01:23:15Z

In storage vault mode during BE startup, if an aws role arn s3 vault is detected, we will check if the machine which BE deployed has been configured correctly. it will not allow BE starting if configuration is incorrect.

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-05-23T01:23:20Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

swjtu-zhanglei · 2025-05-23T01:23:24Z

run buildall

doris-robot · 2025-05-23T02:09:48Z

TPC-H: Total hot run time: 33831 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d85e6a13bd04d886a21e8b8fe0bc918677cfc06e, data reload: false

------ Round 1 ----------------------------------
q1	26425	5103	5033	5033
q2	2080	280	191	191
q3	10390	1228	727	727
q4	10222	1030	524	524
q5	7536	2262	2383	2262
q6	184	167	134	134
q7	897	774	607	607
q8	9302	1299	1102	1102
q9	6830	5052	5034	5034
q10	6842	2295	1909	1909
q11	496	285	278	278
q12	354	364	216	216
q13	17759	3664	3104	3104
q14	227	234	222	222
q15	532	483	485	483
q16	436	432	388	388
q17	600	872	358	358
q18	7529	7166	7095	7095
q19	1231	942	597	597
q20	345	350	230	230
q21	3853	3166	2380	2380
q22	1082	1007	957	957
Total cold run time: 115152 ms
Total hot run time: 33831 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5080	5024	5018	5018
q2	233	336	232	232
q3	2144	2596	2322	2322
q4	1366	1797	1319	1319
q5	4430	4448	4400	4400
q6	210	172	127	127
q7	2021	1935	1773	1773
q8	2589	2806	2502	2502
q9	7245	7164	6931	6931
q10	3045	3222	2790	2790
q11	579	528	486	486
q12	698	807	641	641
q13	3551	3899	3344	3344
q14	293	309	267	267
q15	518	485	497	485
q16	463	495	432	432
q17	1144	1577	1367	1367
q18	7793	7402	7405	7402
q19	861	839	898	839
q20	1992	2029	1941	1941
q21	4837	4473	4419	4419
q22	1104	1058	1031	1031
Total cold run time: 52196 ms
Total hot run time: 50068 ms

doris-robot · 2025-05-23T02:22:00Z

TPC-DS: Total hot run time: 192368 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d85e6a13bd04d886a21e8b8fe0bc918677cfc06e, data reload: false

query1	1452	1084	1091	1084
query2	6464	1817	1805	1805
query3	11010	4485	4492	4485
query4	53478	25691	23159	23159
query5	5023	532	449	449
query6	340	199	201	199
query7	4902	484	287	287
query8	305	249	222	222
query9	5602	2621	2606	2606
query10	408	325	282	282
query11	15025	15051	14839	14839
query12	153	111	105	105
query13	1017	528	404	404
query14	10280	6324	6274	6274
query15	204	191	173	173
query16	7094	633	501	501
query17	1089	744	603	603
query18	1576	408	298	298
query19	193	192	170	170
query20	129	131	126	126
query21	206	129	110	110
query22	4431	4422	4257	4257
query23	34230	33561	33708	33561
query24	6475	2420	2448	2420
query25	474	476	420	420
query26	678	275	158	158
query27	2219	487	348	348
query28	3088	2149	2127	2127
query29	577	560	425	425
query30	272	227	192	192
query31	872	841	806	806
query32	82	69	66	66
query33	487	363	324	324
query34	775	872	545	545
query35	793	835	787	787
query36	974	1021	918	918
query37	121	100	80	80
query38	4167	4281	4181	4181
query39	1547	1480	1436	1436
query40	215	121	107	107
query41	58	54	52	52
query42	121	117	119	117
query43	518	513	483	483
query44	1343	833	826	826
query45	187	175	168	168
query46	832	1014	652	652
query47	1868	1910	1869	1869
query48	405	413	327	327
query49	678	510	430	430
query50	697	713	422	422
query51	4229	4278	4153	4153
query52	116	110	98	98
query53	227	255	185	185
query54	597	582	519	519
query55	84	87	87	87
query56	311	312	305	305
query57	1183	1217	1148	1148
query58	274	271	262	262
query59	2700	2755	2675	2675
query60	339	340	321	321
query61	129	127	136	127
query62	718	761	703	703
query63	229	190	197	190
query64	1463	1076	778	778
query65	4256	4258	4220	4220
query66	788	419	315	315
query67	16074	15524	15592	15524
query68	6861	876	521	521
query69	541	318	275	275
query70	1246	1088	1116	1088
query71	460	321	300	300
query72	5905	4835	5222	4835
query73	1274	682	354	354
query74	9258	9237	8613	8613
query75	3341	3197	2681	2681
query76	3828	1174	767	767
query77	544	374	283	283
query78	10188	10093	9268	9268
query79	2971	799	576	576
query80	756	495	461	461
query81	487	250	221	221
query82	535	125	94	94
query83	355	245	246	245
query84	297	103	89	89
query85	788	359	301	301
query86	396	305	277	277
query87	4335	4433	4268	4268
query88	3580	2304	2289	2289
query89	400	318	285	285
query90	1755	203	206	203
query91	142	140	113	113
query92	79	56	56	56
query93	2524	908	573	573
query94	711	404	306	306
query95	367	288	287	287
query96	486	567	278	278
query97	2688	2753	2668	2668
query98	241	202	209	202
query99	1435	1389	1296	1296
Total cold run time: 297625 ms
Total hot run time: 192368 ms

doris-robot · 2025-05-23T02:27:10Z

ClickBench: Total hot run time: 29.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d85e6a13bd04d886a21e8b8fe0bc918677cfc06e, data reload: false

query1	0.04	0.04	0.03
query2	0.14	0.10	0.12
query3	0.26	0.19	0.20
query4	1.60	0.19	0.19
query5	0.45	0.47	0.44
query6	1.16	0.66	0.67
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.59	0.51	0.53
query10	0.56	0.57	0.56
query11	0.15	0.11	0.11
query12	0.16	0.11	0.11
query13	0.61	0.59	0.60
query14	0.79	0.81	0.81
query15	0.87	0.87	0.87
query16	0.39	0.40	0.36
query17	1.03	1.05	1.05
query18	0.22	0.21	0.21
query19	1.92	1.77	1.82
query20	0.02	0.01	0.01
query21	15.38	0.94	0.56
query22	0.76	1.10	0.87
query23	14.71	1.42	0.61
query24	6.76	1.26	0.57
query25	0.47	0.14	0.12
query26	0.58	0.16	0.14
query27	0.05	0.06	0.04
query28	9.84	0.96	0.44
query29	12.60	4.09	3.34
query30	0.25	0.09	0.06
query31	2.83	0.61	0.38
query32	3.24	0.56	0.48
query33	3.08	3.06	3.12
query34	15.70	5.08	4.48
query35	4.54	4.53	4.51
query36	0.66	0.51	0.50
query37	0.09	0.07	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.18	0.13	0.13
query41	0.08	0.03	0.03
query42	0.04	0.02	0.03
query43	0.03	0.03	0.02
Total cold run time: 102.99 s
Total hot run time: 29.11 s

swjtu-zhanglei · 2025-05-23T08:07:38Z

run buildall

doris-robot · 2025-05-23T08:50:24Z

TPC-H: Total hot run time: 33893 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c423f1a3592f594f2ce3cb923d776ba493682339, data reload: false

------ Round 1 ----------------------------------
q1	26190	5106	4993	4993
q2	2079	300	204	204
q3	10360	1251	727	727
q4	10234	994	527	527
q5	7538	2336	2335	2335
q6	178	164	135	135
q7	894	728	616	616
q8	9317	1267	1079	1079
q9	6889	5066	5059	5059
q10	6881	2323	1910	1910
q11	493	287	274	274
q12	351	348	224	224
q13	17810	3699	3066	3066
q14	228	233	214	214
q15	526	481	477	477
q16	429	432	382	382
q17	622	861	366	366
q18	7562	7219	7167	7167
q19	1668	959	570	570
q20	344	325	219	219
q21	3996	3171	2409	2409
q22	1044	1004	940	940
Total cold run time: 115633 ms
Total hot run time: 33893 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5178	5117	5095	5095
q2	238	323	232	232
q3	2146	2676	2325	2325
q4	1374	1782	1374	1374
q5	4497	4416	4342	4342
q6	215	166	129	129
q7	2009	1972	1747	1747
q8	2604	2520	2474	2474
q9	7328	7281	7197	7197
q10	3026	3219	2777	2777
q11	588	501	500	500
q12	690	795	638	638
q13	3520	3878	3327	3327
q14	314	301	268	268
q15	519	486	472	472
q16	419	484	455	455
q17	1156	1523	1400	1400
q18	7737	7561	7579	7561
q19	825	882	915	882
q20	1980	2007	1829	1829
q21	4913	4449	4471	4449
q22	1091	1039	989	989
Total cold run time: 52367 ms
Total hot run time: 50462 ms

doris-robot · 2025-05-23T09:02:29Z

TPC-DS: Total hot run time: 192794 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c423f1a3592f594f2ce3cb923d776ba493682339, data reload: false

query1	1416	1093	1031	1031
query2	6320	1822	1778	1778
query3	11020	4392	4431	4392
query4	52371	24633	23633	23633
query5	4945	553	478	478
query6	362	209	200	200
query7	4881	516	298	298
query8	311	253	241	241
query9	5370	2674	2649	2649
query10	431	324	295	295
query11	15111	15019	14827	14827
query12	163	113	109	109
query13	1026	556	406	406
query14	10057	6348	6265	6265
query15	209	213	178	178
query16	7145	683	512	512
query17	1061	731	572	572
query18	1566	409	306	306
query19	198	192	174	174
query20	126	135	119	119
query21	203	125	107	107
query22	4387	4558	4414	4414
query23	34415	33459	33566	33459
query24	6585	2492	2456	2456
query25	479	508	407	407
query26	696	283	155	155
query27	2313	521	341	341
query28	2959	2165	2169	2165
query29	615	588	446	446
query30	274	224	197	197
query31	852	879	805	805
query32	76	63	64	63
query33	441	376	330	330
query34	802	862	544	544
query35	791	841	756	756
query36	952	974	904	904
query37	116	98	73	73
query38	4338	4234	4205	4205
query39	1510	1534	1478	1478
query40	254	116	110	110
query41	56	52	58	52
query42	126	112	124	112
query43	512	515	488	488
query44	1397	846	832	832
query45	182	177	176	176
query46	913	1049	648	648
query47	1852	1914	1786	1786
query48	407	435	332	332
query49	759	523	448	448
query50	673	691	421	421
query51	4348	4329	4107	4107
query52	112	110	97	97
query53	234	263	191	191
query54	588	584	524	524
query55	87	87	82	82
query56	297	313	317	313
query57	1208	1201	1136	1136
query58	276	290	267	267
query59	2666	2798	2612	2612
query60	339	323	338	323
query61	129	156	128	128
query62	765	723	695	695
query63	225	188	191	188
query64	1486	1042	688	688
query65	4339	4207	4216	4207
query66	726	398	302	302
query67	15826	15502	15597	15502
query68	5510	857	524	524
query69	510	304	273	273
query70	1210	1119	1116	1116
query71	426	325	286	286
query72	5784	4837	5020	4837
query73	1338	710	352	352
query74	8905	9072	8704	8704
query75	3232	3195	2678	2678
query76	3724	1199	776	776
query77	561	403	295	295
query78	9958	10131	9359	9359
query79	1951	822	588	588
query80	827	498	447	447
query81	511	251	227	227
query82	420	133	95	95
query83	290	257	229	229
query84	340	110	83	83
query85	762	356	309	309
query86	332	299	288	288
query87	4404	4460	4314	4314
query88	2960	2275	2259	2259
query89	414	314	283	283
query90	1601	218	200	200
query91	145	147	111	111
query92	63	63	55	55
query93	1731	916	588	588
query94	725	413	316	316
query95	371	289	312	289
query96	497	557	280	280
query97	2683	2777	2606	2606
query98	231	207	205	205
query99	1327	1383	1287	1287
Total cold run time: 291418 ms
Total hot run time: 192794 ms

doris-robot · 2025-05-23T09:07:39Z

ClickBench: Total hot run time: 28.99 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c423f1a3592f594f2ce3cb923d776ba493682339, data reload: false

query1	0.04	0.04	0.04
query2	0.13	0.10	0.13
query3	0.25	0.19	0.19
query4	1.59	0.19	0.19
query5	0.44	0.42	0.43
query6	1.14	0.66	0.66
query7	0.03	0.01	0.02
query8	0.05	0.04	0.04
query9	0.60	0.52	0.52
query10	0.57	0.58	0.57
query11	0.16	0.11	0.11
query12	0.14	0.12	0.11
query13	0.61	0.59	0.58
query14	0.78	0.81	0.81
query15	0.86	0.86	0.84
query16	0.39	0.39	0.38
query17	1.05	1.05	1.07
query18	0.23	0.21	0.21
query19	1.95	1.88	1.82
query20	0.01	0.02	0.01
query21	15.39	0.94	0.55
query22	0.76	1.13	0.60
query23	15.07	1.38	0.56
query24	7.14	1.16	0.93
query25	0.48	0.20	0.08
query26	0.58	0.17	0.15
query27	0.05	0.05	0.05
query28	10.29	0.91	0.47
query29	12.54	3.97	3.32
query30	0.26	0.09	0.06
query31	2.82	0.62	0.39
query32	3.24	0.56	0.45
query33	3.05	3.09	3.02
query34	15.72	5.11	4.46
query35	4.50	4.54	4.46
query36	0.65	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.14
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 104.02 s
Total hot run time: 28.99 s

hello-stephen · 2025-05-23T10:06:37Z

BE UT Coverage Report

Increment line coverage 2.94% (1/34) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	56.00% (14920/26643)
Line Coverage	44.79% (132733/296316)
Region Coverage	43.86% (66720/152129)
Branch Coverage	38.49% (34228/88928)

hello-stephen · 2025-05-23T10:58:31Z

BE Regression && UT Coverage Report

Increment line coverage 2.94% (1/34) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.52% (20856/26226)
Line Coverage	72.69% (215382/296303)
Region Coverage	70.85% (126589/178673)
Branch Coverage	64.61% (65623/101566)

swjtu-zhanglei · 2025-05-28T07:00:02Z

run buildall

doris-robot · 2025-05-28T08:34:41Z

TPC-H: Total hot run time: 33682 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 63c178fb54accab9b9d279d9f00273fa2b5da4a6, data reload: false

------ Round 1 ----------------------------------
q1	26161	5019	5022	5019
q2	2072	274	183	183
q3	10692	1268	711	711
q4	10247	1014	545	545
q5	8154	2400	2306	2306
q6	184	164	135	135
q7	893	749	596	596
q8	9315	1297	1087	1087
q9	6842	5035	5102	5035
q10	6853	2318	1892	1892
q11	484	299	276	276
q12	340	352	212	212
q13	17766	3650	3088	3088
q14	241	239	214	214
q15	528	492	481	481
q16	429	444	379	379
q17	604	860	357	357
q18	7787	7114	7131	7114
q19	1727	954	555	555
q20	330	343	220	220
q21	3623	2568	2301	2301
q22	1045	1029	976	976
Total cold run time: 116317 ms
Total hot run time: 33682 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5201	5058	5109	5058
q2	235	323	231	231
q3	2117	2645	2266	2266
q4	1366	1764	1336	1336
q5	4523	4422	4387	4387
q6	213	168	128	128
q7	2001	1913	1760	1760
q8	2592	2503	2497	2497
q9	7330	7102	6934	6934
q10	2997	3234	2766	2766
q11	645	520	495	495
q12	697	763	598	598
q13	3503	3953	3220	3220
q14	271	287	277	277
q15	525	480	487	480
q16	453	543	445	445
q17	1129	1557	1385	1385
q18	7796	7648	7464	7464
q19	792	824	828	824
q20	1993	2064	1892	1892
q21	4674	4552	4380	4380
q22	1106	1057	1052	1052
Total cold run time: 52159 ms
Total hot run time: 49875 ms

doris-robot · 2025-05-28T08:47:01Z

TPC-DS: Total hot run time: 192980 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 63c178fb54accab9b9d279d9f00273fa2b5da4a6, data reload: false

query1	1421	1096	1083	1083
query2	6254	1894	1889	1889
query3	11018	4599	4517	4517
query4	53644	25960	23379	23379
query5	5210	524	457	457
query6	353	204	192	192
query7	4968	505	293	293
query8	325	249	240	240
query9	6261	2650	2662	2650
query10	441	334	277	277
query11	15172	15004	14745	14745
query12	170	122	99	99
query13	1115	536	414	414
query14	10141	6300	6360	6300
query15	206	200	176	176
query16	7128	679	515	515
query17	1101	768	610	610
query18	1550	422	329	329
query19	208	200	175	175
query20	132	129	125	125
query21	207	135	112	112
query22	4273	4419	4257	4257
query23	34516	33629	33472	33472
query24	6766	2438	2427	2427
query25	502	475	416	416
query26	715	275	156	156
query27	2384	508	350	350
query28	3225	2177	2202	2177
query29	571	583	423	423
query30	279	225	187	187
query31	873	842	755	755
query32	73	61	64	61
query33	480	406	299	299
query34	799	876	545	545
query35	795	826	740	740
query36	946	992	898	898
query37	110	94	77	77
query38	4266	4245	4194	4194
query39	1528	1473	1445	1445
query40	215	118	108	108
query41	59	57	54	54
query42	135	117	122	117
query43	524	522	501	501
query44	1333	853	837	837
query45	184	173	164	164
query46	862	1031	657	657
query47	1850	1874	1777	1777
query48	394	428	326	326
query49	705	484	433	433
query50	659	703	426	426
query51	4188	4252	4185	4185
query52	108	107	98	98
query53	230	261	192	192
query54	583	562	516	516
query55	86	89	83	83
query56	304	289	291	289
query57	1197	1197	1136	1136
query58	271	268	250	250
query59	2844	2851	2721	2721
query60	343	332	306	306
query61	128	139	125	125
query62	699	753	670	670
query63	227	190	204	190
query64	1954	1036	678	678
query65	4316	4286	4240	4240
query66	743	389	299	299
query67	15949	15648	15309	15309
query68	6778	890	539	539
query69	553	313	321	313
query70	1218	1104	1137	1104
query71	518	317	290	290
query72	5834	4799	4953	4799
query73	1364	777	361	361
query74	8909	9099	8975	8975
query75	3852	3217	2737	2737
query76	4369	1187	756	756
query77	637	354	287	287
query78	9967	10273	9557	9557
query79	6381	772	563	563
query80	679	524	440	440
query81	479	265	215	215
query82	676	133	95	95
query83	332	260	251	251
query84	291	103	93	93
query85	786	345	314	314
query86	356	314	309	309
query87	4364	4509	4325	4325
query88	3209	2287	2269	2269
query89	457	317	290	290
query90	1943	211	214	211
query91	143	142	108	108
query92	78	59	53	53
query93	3067	941	597	597
query94	688	408	312	312
query95	363	299	286	286
query96	513	564	287	287
query97	2725	2786	2632	2632
query98	218	208	197	197
query99	1440	1424	1272	1272
Total cold run time: 304403 ms
Total hot run time: 192980 ms

doris-robot · 2025-05-28T08:52:12Z

ClickBench: Total hot run time: 29.71 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 63c178fb54accab9b9d279d9f00273fa2b5da4a6, data reload: false

query1	0.04	0.04	0.04
query2	0.13	0.10	0.12
query3	0.26	0.20	0.19
query4	1.60	0.19	0.20
query5	0.46	0.45	0.44
query6	1.17	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.59	0.51	0.51
query10	0.56	0.58	0.57
query11	0.16	0.10	0.11
query12	0.15	0.12	0.12
query13	0.61	0.59	0.60
query14	0.79	0.80	0.80
query15	0.88	0.83	0.87
query16	0.38	0.38	0.38
query17	1.04	1.07	1.03
query18	0.23	0.22	0.21
query19	1.86	1.87	1.79
query20	0.02	0.02	0.01
query21	15.40	0.91	0.53
query22	0.75	1.24	1.02
query23	14.68	1.37	0.63
query24	6.72	1.32	1.15
query25	0.51	0.21	0.08
query26	0.58	0.17	0.14
query27	0.05	0.05	0.04
query28	10.33	0.92	0.45
query29	12.52	3.99	3.31
query30	0.26	0.09	0.06
query31	2.83	0.58	0.39
query32	3.23	0.56	0.48
query33	3.09	3.03	3.08
query34	15.75	5.11	4.48
query35	4.54	4.55	4.52
query36	0.68	0.50	0.48
query37	0.09	0.07	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.15	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.39 s
Total hot run time: 29.71 s

swjtu-zhanglei · 2025-05-30T01:49:12Z

run buildall

doris-robot · 2025-05-30T02:35:03Z

TPC-H: Total hot run time: 33645 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fa2b1814391491587600cb0e5cee4de206a6302f, data reload: false

------ Round 1 ----------------------------------
q1	26128	4991	4965	4965
q2	1960	278	178	178
q3	10508	1260	710	710
q4	10217	999	506	506
q5	7569	2410	2338	2338
q6	180	168	131	131
q7	931	732	605	605
q8	9302	1254	1040	1040
q9	6871	5124	5157	5124
q10	6831	2298	1901	1901
q11	471	294	280	280
q12	349	345	222	222
q13	17772	3655	3078	3078
q14	235	227	217	217
q15	562	486	487	486
q16	427	433	375	375
q17	598	862	363	363
q18	7542	7261	7079	7079
q19	1521	956	556	556
q20	321	319	215	215
q21	3798	3189	2323	2323
q22	1046	960	953	953
Total cold run time: 115139 ms
Total hot run time: 33645 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5151	5098	5045	5045
q2	230	310	228	228
q3	2160	2670	2326	2326
q4	1349	1804	1354	1354
q5	4527	4375	4414	4375
q6	219	171	130	130
q7	1994	1937	1783	1783
q8	2599	2553	2501	2501
q9	7191	7297	6954	6954
q10	3017	3193	2767	2767
q11	579	522	499	499
q12	677	786	607	607
q13	3475	3900	3274	3274
q14	288	300	261	261
q15	528	472	487	472
q16	463	511	461	461
q17	1143	1531	1421	1421
q18	7757	7464	7486	7464
q19	804	838	806	806
q20	2032	2040	1908	1908
q21	4791	4338	4482	4338
q22	1055	1068	998	998
Total cold run time: 52029 ms
Total hot run time: 49972 ms

doris-robot · 2025-05-30T02:47:28Z

TPC-DS: Total hot run time: 193220 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fa2b1814391491587600cb0e5cee4de206a6302f, data reload: false

query1	1400	1108	1063	1063
query2	6106	1827	1819	1819
query3	11016	4589	4654	4589
query4	57320	25091	23334	23334
query5	5049	483	450	450
query6	342	211	209	209
query7	4931	504	288	288
query8	295	244	215	215
query9	5687	2636	2646	2636
query10	436	326	275	275
query11	15179	14975	14854	14854
query12	161	107	104	104
query13	1063	539	412	412
query14	10203	6463	6491	6463
query15	216	204	182	182
query16	7116	637	498	498
query17	1077	755	601	601
query18	1656	420	320	320
query19	223	206	178	178
query20	137	124	124	124
query21	211	128	107	107
query22	4384	4263	4330	4263
query23	34120	33459	33541	33459
query24	6840	2461	2445	2445
query25	486	465	399	399
query26	686	276	170	170
query27	2235	534	345	345
query28	3037	2203	2177	2177
query29	584	548	425	425
query30	280	226	192	192
query31	875	936	789	789
query32	74	60	64	60
query33	441	383	317	317
query34	771	861	529	529
query35	798	835	775	775
query36	953	987	903	903
query37	112	108	76	76
query38	4347	4311	4250	4250
query39	1556	1473	1484	1473
query40	206	125	111	111
query41	59	58	56	56
query42	129	117	118	117
query43	505	510	494	494
query44	1412	867	857	857
query45	181	178	174	174
query46	856	1039	664	664
query47	1869	1863	1803	1803
query48	413	437	331	331
query49	653	504	422	422
query50	681	709	407	407
query51	4220	4229	4248	4229
query52	122	113	109	109
query53	235	254	182	182
query54	583	581	514	514
query55	86	87	84	84
query56	303	328	306	306
query57	1173	1243	1113	1113
query58	281	276	260	260
query59	2691	2829	2643	2643
query60	346	326	310	310
query61	150	127	125	125
query62	739	726	718	718
query63	221	186	188	186
query64	1492	1019	682	682
query65	4226	4171	4151	4151
query66	695	415	297	297
query67	15908	15766	15511	15511
query68	7965	896	532	532
query69	558	305	267	267
query70	1196	1106	1114	1106
query71	509	340	300	300
query72	5857	4799	5018	4799
query73	1459	671	353	353
query74	9278	9062	8823	8823
query75	3797	3214	2761	2761
query76	4136	1212	751	751
query77	612	385	284	284
query78	10129	10070	9386	9386
query79	6798	792	559	559
query80	687	534	452	452
query81	477	261	221	221
query82	726	135	95	95
query83	350	256	251	251
query84	297	134	90	90
query85	812	364	320	320
query86	400	305	282	282
query87	4504	4557	4308	4308
query88	4108	2323	2366	2323
query89	459	318	292	292
query90	1893	217	214	214
query91	139	146	113	113
query92	77	60	57	57
query93	3929	934	576	576
query94	689	407	290	290
query95	370	306	288	288
query96	514	565	280	280
query97	2761	2776	2682	2682
query98	278	198	204	198
query99	1442	1394	1343	1343
Total cold run time: 310044 ms
Total hot run time: 193220 ms

doris-robot · 2025-05-30T02:52:48Z

ClickBench: Total hot run time: 29.49 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fa2b1814391491587600cb0e5cee4de206a6302f, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.10	0.12
query3	0.26	0.20	0.20
query4	1.59	0.20	0.11
query5	0.43	0.42	0.42
query6	1.16	0.67	0.65
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.61	0.51	0.50
query10	0.57	0.58	0.57
query11	0.17	0.11	0.11
query12	0.15	0.12	0.11
query13	0.63	0.61	0.60
query14	0.79	0.81	0.82
query15	0.88	0.88	0.88
query16	0.38	0.38	0.37
query17	1.09	1.08	1.07
query18	0.23	0.21	0.21
query19	1.92	1.91	1.85
query20	0.01	0.01	0.02
query21	15.39	0.89	0.53
query22	0.76	1.16	0.59
query23	15.06	1.36	0.64
query24	6.77	1.38	1.22
query25	0.48	0.14	0.14
query26	0.67	0.16	0.13
query27	0.05	0.05	0.04
query28	8.98	0.85	0.44
query29	12.57	4.04	3.40
query30	0.25	0.09	0.07
query31	2.81	0.58	0.39
query32	3.22	0.55	0.46
query33	3.10	3.07	3.08
query34	15.77	5.04	4.45
query35	4.53	4.52	4.48
query36	0.67	0.50	0.48
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.18	0.15	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.02	0.03
Total cold run time: 102.69 s
Total hot run time: 29.49 s

hello-stephen · 2025-05-30T03:57:38Z

BE UT Coverage Report

Increment line coverage 2.94% (1/34) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	56.16% (15028/26759)
Line Coverage	45.07% (134149/297645)
Region Coverage	44.19% (67508/152784)
Branch Coverage	38.75% (34578/89244)

hello-stephen · 2025-05-30T05:39:21Z

BE Regression && UT Coverage Report

Increment line coverage 2.94% (1/34) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.52% (20947/26343)
Line Coverage	72.69% (216355/297634)
Region Coverage	70.87% (127348/179689)
Branch Coverage	64.60% (65924/102044)

swjtu-zhanglei · 2025-06-03T01:51:47Z

run buildall

doris-robot · 2025-06-03T02:37:42Z

TPC-H: Total hot run time: 33862 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 307645612cf505c008e38be990d8cc6ee253926f, data reload: false

------ Round 1 ----------------------------------
q1	25998	5464	5003	5003
q2	1968	272	187	187
q3	10282	1239	714	714
q4	10228	1012	517	517
q5	7561	2398	2338	2338
q6	190	160	132	132
q7	894	735	617	617
q8	9304	1265	1059	1059
q9	6820	5120	5062	5062
q10	6866	2317	1887	1887
q11	500	288	277	277
q12	352	347	220	220
q13	17774	3690	3094	3094
q14	235	232	209	209
q15	554	488	486	486
q16	425	430	376	376
q17	588	869	361	361
q18	7602	7250	7158	7158
q19	1617	942	558	558
q20	344	331	218	218
q21	3714	2595	2412	2412
q22	1040	1034	977	977
Total cold run time: 114856 ms
Total hot run time: 33862 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5204	5073	5060	5060
q2	248	338	227	227
q3	2182	2652	2313	2313
q4	1348	1778	1386	1386
q5	4411	4415	4397	4397
q6	209	178	133	133
q7	2041	1935	1782	1782
q8	2601	2592	2578	2578
q9	7188	7191	6933	6933
q10	3081	3213	2816	2816
q11	589	520	487	487
q12	683	771	658	658
q13	3502	3884	3232	3232
q14	284	296	302	296
q15	530	492	471	471
q16	449	473	442	442
q17	1137	1577	1398	1398
q18	7803	7646	7452	7452
q19	816	774	855	774
q20	2074	2043	1860	1860
q21	4778	4459	4525	4459
q22	1144	1077	1020	1020
Total cold run time: 52302 ms
Total hot run time: 50174 ms

doris-robot · 2025-06-03T02:49:42Z

TPC-DS: Total hot run time: 193134 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 307645612cf505c008e38be990d8cc6ee253926f, data reload: false

query1	1407	1089	1098	1089
query2	6427	1826	1855	1826
query3	11134	4623	4585	4585
query4	25526	24045	23463	23463
query5	5024	621	466	466
query6	305	220	207	207
query7	3990	523	282	282
query8	263	223	212	212
query9	8488	2634	2648	2634
query10	491	322	275	275
query11	15320	15051	14778	14778
query12	162	115	107	107
query13	1567	548	410	410
query14	8739	6314	6378	6314
query15	208	198	170	170
query16	7349	667	496	496
query17	1166	807	616	616
query18	2078	426	328	328
query19	214	222	177	177
query20	128	128	121	121
query21	209	126	114	114
query22	4397	4644	4349	4349
query23	34600	33819	33621	33621
query24	8639	2463	2468	2463
query25	526	474	415	415
query26	1242	271	214	214
query27	2745	517	340	340
query28	4793	2189	2160	2160
query29	748	583	449	449
query30	272	236	193	193
query31	914	852	787	787
query32	77	71	67	67
query33	539	388	308	308
query34	818	887	537	537
query35	803	850	742	742
query36	962	999	933	933
query37	114	99	76	76
query38	4232	4268	4338	4268
query39	1499	1473	1451	1451
query40	217	126	112	112
query41	62	56	81	56
query42	126	115	114	114
query43	501	517	505	505
query44	1372	826	835	826
query45	183	182	172	172
query46	891	1034	654	654
query47	1840	1885	1798	1798
query48	387	444	322	322
query49	756	487	396	396
query50	681	691	418	418
query51	4134	4341	4190	4190
query52	116	112	104	104
query53	231	272	189	189
query54	611	598	535	535
query55	90	93	91	91
query56	314	346	299	299
query57	1173	1211	1125	1125
query58	273	265	257	257
query59	2759	2944	2732	2732
query60	366	362	316	316
query61	131	142	126	126
query62	806	726	689	689
query63	234	204	197	197
query64	4125	1121	707	707
query65	4404	4240	4185	4185
query66	1016	398	309	309
query67	15915	15818	15308	15308
query68	7739	900	520	520
query69	456	304	269	269
query70	1186	1122	1114	1114
query71	465	320	308	308
query72	5841	4715	4808	4715
query73	561	629	358	358
query74	9195	9117	8937	8937
query75	3929	3231	2700	2700
query76	3102	1192	786	786
query77	759	390	285	285
query78	10252	9991	9321	9321
query79	6211	852	559	559
query80	690	518	444	444
query81	480	257	221	221
query82	691	132	100	100
query83	291	245	238	238
query84	293	119	89	89
query85	787	351	310	310
query86	340	312	276	276
query87	4415	4467	4327	4327
query88	3900	2291	2261	2261
query89	457	318	280	280
query90	1979	211	212	211
query91	144	142	111	111
query92	73	58	59	58
query93	3695	942	581	581
query94	678	420	297	297
query95	375	294	293	293
query96	500	593	278	278
query97	2721	2846	2628	2628
query98	228	211	225	211
query99	1424	1398	1273	1273
Total cold run time: 285686 ms
Total hot run time: 193134 ms

doris-robot · 2025-06-03T02:54:53Z

ClickBench: Total hot run time: 29.2 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 307645612cf505c008e38be990d8cc6ee253926f, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.10	0.12
query3	0.25	0.20	0.20
query4	1.59	0.19	0.21
query5	0.47	0.46	0.45
query6	1.16	0.67	0.65
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.58	0.51	0.53
query10	0.58	0.56	0.57
query11	0.16	0.11	0.12
query12	0.14	0.12	0.12
query13	0.61	0.59	0.60
query14	0.79	0.80	0.81
query15	0.88	0.84	0.85
query16	0.37	0.37	0.38
query17	1.05	1.06	1.04
query18	0.24	0.21	0.22
query19	1.92	1.78	1.82
query20	0.02	0.02	0.01
query21	15.40	0.92	0.54
query22	0.75	1.06	0.65
query23	15.12	1.44	0.65
query24	7.23	1.82	0.81
query25	0.47	0.16	0.08
query26	0.55	0.16	0.15
query27	0.05	0.04	0.05
query28	8.87	0.87	0.45
query29	12.59	4.08	3.36
query30	0.26	0.09	0.07
query31	2.81	0.60	0.38
query32	3.23	0.55	0.46
query33	3.00	3.08	3.10
query34	15.81	5.23	4.53
query35	4.58	4.55	4.51
query36	0.68	0.49	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.16	0.14	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.05	0.04	0.03
Total cold run time: 102.94 s
Total hot run time: 29.2 s

hello-stephen · 2025-06-03T03:53:02Z

BE UT Coverage Report

Increment line coverage 2.94% (1/34) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	56.05% (15021/26801)
Line Coverage	44.91% (134096/298570)
Region Coverage	44.03% (67464/153231)
Branch Coverage	38.58% (34553/89560)

doris-robot · 2025-06-24T04:34:35Z

BE UT Coverage Report

Increment line coverage 2.86% (1/35) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	56.63% (15126/26712)
Line Coverage	45.41% (135589/298596)
Region Coverage	44.57% (68236/153094)
Branch Coverage	39.15% (35064/89556)

hello-stephen · 2025-06-24T06:02:36Z

BE Regression && UT Coverage Report

Increment line coverage 2.86% (1/35) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	62.01% (16297/26280)
Line Coverage	51.53% (153763/298414)
Region Coverage	48.93% (88109/180076)
Branch Coverage	42.40% (43384/102332)

dataroaring

LGTM

github-actions · 2025-06-25T12:44:07Z

PR approved by at least one committer and no changes requested.

…arting (#51175) In storage vault mode during `BE` startup, if an aws role arn s3 vault is detected, we will check if the machine which `BE` deployed has been configured correctly. it will not allow `BE` starting if configuration is incorrect.

…r BE when starting #51175 (#52319) Cherry-picked from #51175 Co-authored-by: Lei Zhang <zhanglei@selectdb.com>

…r BE when starting #51175 (#52320) Cherry-picked from #51175 Co-authored-by: Lei Zhang <zhanglei@selectdb.com>

* apache#51175 introduced a deadlock problem: 1. the `sync_storage_vault()` func wait for heartbeat thread to set config::meta_service_endpoint 2. the heartbeat thread wait for the `sync_storage_vault()` func success to start

* #51175 introduced a deadlock problem: 1. the `sync_storage_vault()` func wait for heartbeat thread to set config::meta_service_endpoint 2. the heartbeat thread wait for the `sync_storage_vault()` func success to start

…r BE when starting apache#51175 (apache#52319) Cherry-picked from apache#51175 Co-authored-by: Lei Zhang <zhanglei@selectdb.com>

* apache#51175 introduced a deadlock problem: 1. the `sync_storage_vault()` func wait for heartbeat thread to set config::meta_service_endpoint 2. the heartbeat thread wait for the `sync_storage_vault()` func success to start

swjtu-zhanglei force-pushed the master-fix13 branch from d85e6a1 to c423f1a Compare May 23, 2025 08:06

swjtu-zhanglei marked this pull request as ready for review June 3, 2025 01:51

swjtu-zhanglei requested review from dataroaring and gavinchou as code owners June 3, 2025 01:51

dataroaring approved these changes Jun 25, 2025

View reviewed changes

gavinchou approved these changes Jun 25, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 25, 2025

dataroaring added the dev/3.1.x label Jun 25, 2025

dataroaring merged commit c50992b into apache:master Jun 25, 2025
25 of 28 checks passed

github-actions bot mentioned this pull request Jun 25, 2025

branch-3.0: [feat](storage vault) Check storage vault connectivity for BE when starting #51175 #52319

Merged

github-actions bot mentioned this pull request Jun 25, 2025

branch-3.1: [feat](storage vault) Check storage vault connectivity for BE when starting #51175 #52320

Merged

dataroaring pushed a commit that referenced this pull request Jun 26, 2025

branch-3.0: [feat](storage vault) Check storage vault connectivity fo…

8710bea

…r BE when starting #51175 (#52319) Cherry-picked from #51175 Co-authored-by: Lei Zhang <zhanglei@selectdb.com>

dataroaring added dev/3.0.7-merged and removed dev/3.0.x labels Jun 26, 2025

gavinchou added the cloud label Jun 26, 2025

morrySnow pushed a commit that referenced this pull request Jun 27, 2025

branch-3.1: [feat](storage vault) Check storage vault connectivity fo…

2ae4983

…r BE when starting #51175 (#52320) Cherry-picked from #51175 Co-authored-by: Lei Zhang <zhanglei@selectdb.com>

morrySnow added dev/3.1.0-merged and removed dev/3.1.x labels Jun 27, 2025

swjtu-zhanglei mentioned this pull request Jun 30, 2025

[fix](be) Fix check_storage_vault deadlock #52541

Merged

16 tasks

gavinchou mentioned this pull request Aug 15, 2025

Release Note 3.0.7 #54868

Open

[feat](storage vault) Check storage vault connectivity for BE when starting #51175

[feat](storage vault) Check storage vault connectivity for BE when starting #51175

Uh oh!

Conversation

swjtu-zhanglei commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented May 23, 2025

Uh oh!

swjtu-zhanglei commented May 23, 2025

Uh oh!

doris-robot commented May 23, 2025

Uh oh!

doris-robot commented May 23, 2025

Uh oh!

doris-robot commented May 23, 2025

Uh oh!

swjtu-zhanglei commented May 23, 2025

Uh oh!

doris-robot commented May 23, 2025

Uh oh!

doris-robot commented May 23, 2025

Uh oh!

doris-robot commented May 23, 2025

Uh oh!

hello-stephen commented May 23, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented May 23, 2025

BE Regression && UT Coverage Report

Uh oh!

swjtu-zhanglei commented May 28, 2025

Uh oh!

doris-robot commented May 28, 2025

Uh oh!

doris-robot commented May 28, 2025

Uh oh!

doris-robot commented May 28, 2025

Uh oh!

swjtu-zhanglei commented May 30, 2025

Uh oh!

doris-robot commented May 30, 2025

Uh oh!

doris-robot commented May 30, 2025

Uh oh!

doris-robot commented May 30, 2025

Uh oh!

hello-stephen commented May 30, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented May 30, 2025

BE Regression && UT Coverage Report

Uh oh!

swjtu-zhanglei commented Jun 3, 2025

Uh oh!

doris-robot commented Jun 3, 2025

Uh oh!

doris-robot commented Jun 3, 2025

Uh oh!

doris-robot commented Jun 3, 2025

Uh oh!

hello-stephen commented Jun 3, 2025

BE UT Coverage Report

Uh oh!

doris-robot commented Jun 24, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 24, 2025

BE Regression && UT Coverage Report

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 25, 2025

Uh oh!

Uh oh!

swjtu-zhanglei commented May 23, 2025 •

edited

Loading