[opt](vault) Check hdfs connectivity when creating hdfs storage vault #48369

swjtu-zhanglei · 2025-02-26T08:12:48Z

What problem does this PR solve?

When creating an HDFS storage vault, perform connection and permission checks to avoid outputting error messages that could cause the storage vault to throw errors during usage after creation.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-02-26T08:12:54Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

swjtu-zhanglei · 2025-02-26T08:13:09Z

run buildall

swjtu-zhanglei · 2025-02-26T08:14:04Z

#48351 当前pr依赖48351的bugfix先合入

doris-robot · 2025-02-26T08:36:41Z

TPC-H: Total hot run time: 32072 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 07a9c0d50889d3be1e3a6ad0061d1fa1f49930df, data reload: false

------ Round 1 ----------------------------------
q1	17586	5424	5281	5281
q2	2064	319	171	171
q3	10387	1368	715	715
q4	10223	1063	527	527
q5	7555	2393	2487	2393
q6	211	180	137	137
q7	939	757	606	606
q8	9316	1478	1213	1213
q9	5199	4763	4782	4763
q10	6881	2333	1880	1880
q11	487	277	253	253
q12	352	385	222	222
q13	17753	3812	3060	3060
q14	234	237	214	214
q15	519	461	458	458
q16	621	612	576	576
q17	590	931	343	343
q18	6942	6366	6215	6215
q19	1201	1094	561	561
q20	322	335	191	191
q21	2915	2251	1983	1983
q22	380	345	310	310
Total cold run time: 102677 ms
Total hot run time: 32072 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5534	5451	5439	5439
q2	247	349	231	231
q3	2191	2713	2293	2293
q4	1461	1869	1434	1434
q5	4280	4149	4165	4149
q6	249	173	124	124
q7	1918	1966	1770	1770
q8	2859	2908	2651	2651
q9	7233	7188	7155	7155
q10	3086	3181	2779	2779
q11	605	509	487	487
q12	698	786	610	610
q13	3642	3973	3220	3220
q14	272	290	276	276
q15	508	449	452	449
q16	652	691	647	647
q17	1205	1752	1345	1345
q18	7643	7542	7338	7338
q19	869	840	923	840
q20	1989	2088	1860	1860
q21	5559	5059	4945	4945
q22	631	579	545	545
Total cold run time: 53331 ms
Total hot run time: 50587 ms

doris-robot · 2025-02-26T08:48:08Z

TPC-DS: Total hot run time: 185016 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 07a9c0d50889d3be1e3a6ad0061d1fa1f49930df, data reload: false

query1	1012	386	383	383
query2	6558	1924	1861	1861
query3	6783	213	215	213
query4	26156	23640	23663	23640
query5	4340	663	533	533
query6	301	205	188	188
query7	4607	496	296	296
query8	299	237	245	237
query9	8653	2607	2604	2604
query10	473	328	256	256
query11	15823	15120	14826	14826
query12	178	110	106	106
query13	1676	549	409	409
query14	10806	7080	6662	6662
query15	225	196	178	178
query16	6993	637	482	482
query17	1605	736	570	570
query18	1244	379	319	319
query19	202	192	165	165
query20	125	120	123	120
query21	210	124	102	102
query22	4229	4408	4172	4172
query23	33923	32975	33087	32975
query24	7327	2390	2403	2390
query25	508	450	380	380
query26	1225	271	159	159
query27	2025	495	329	329
query28	3902	2432	2413	2413
query29	627	556	414	414
query30	241	184	159	159
query31	947	831	777	777
query32	74	67	64	64
query33	562	356	309	309
query34	781	858	495	495
query35	811	819	746	746
query36	949	972	871	871
query37	119	97	77	77
query38	4115	4194	4094	4094
query39	1530	1584	1386	1386
query40	199	116	102	102
query41	55	50	50	50
query42	127	112	107	107
query43	499	488	470	470
query44	1295	780	785	780
query45	181	166	162	162
query46	871	1026	650	650
query47	1721	1770	1687	1687
query48	377	407	301	301
query49	794	502	412	412
query50	683	736	423	423
query51	4186	4165	4150	4150
query52	108	109	95	95
query53	226	255	185	185
query54	497	494	402	402
query55	86	88	84	84
query56	292	248	269	248
query57	1122	1135	1051	1051
query58	275	234	238	234
query59	2499	2715	2806	2715
query60	294	263	259	259
query61	120	118	116	116
query62	825	723	654	654
query63	243	192	197	192
query64	4236	1061	712	712
query65	3190	3107	3128	3107
query66	1055	402	322	322
query67	15889	15258	15402	15258
query68	7815	885	509	509
query69	466	294	266	266
query70	1167	1103	1110	1103
query71	530	297	276	276
query72	5839	3568	3672	3568
query73	780	724	352	352
query74	8949	9230	8721	8721
query75	3777	3178	2667	2667
query76	3702	1178	741	741
query77	781	377	289	289
query78	9921	10140	9379	9379
query79	2098	845	594	594
query80	717	524	503	503
query81	498	273	250	250
query82	558	127	98	98
query83	188	167	155	155
query84	256	93	75	75
query85	755	343	315	315
query86	340	301	286	286
query87	4490	4554	4543	4543
query88	2829	2237	2225	2225
query89	460	326	293	293
query90	2078	190	190	190
query91	141	136	109	109
query92	72	62	59	59
query93	1147	1017	575	575
query94	659	406	274	274
query95	353	269	263	263
query96	481	566	268	268
query97	3314	3395	3292	3292
query98	216	201	203	201
query99	1482	1435	1289	1289
Total cold run time: 270968 ms
Total hot run time: 185016 ms

doris-robot · 2025-02-26T08:53:27Z

ClickBench: Total hot run time: 30.81 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 07a9c0d50889d3be1e3a6ad0061d1fa1f49930df, data reload: false

query1	0.04	0.05	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.60	0.10	0.10
query5	0.59	0.54	0.55
query6	1.19	0.72	0.72
query7	0.03	0.02	0.02
query8	0.04	0.04	0.03
query9	0.60	0.52	0.52
query10	0.58	0.57	0.57
query11	0.16	0.10	0.11
query12	0.15	0.11	0.12
query13	0.61	0.60	0.60
query14	2.69	2.72	2.68
query15	0.91	0.85	0.85
query16	0.38	0.38	0.37
query17	1.00	0.99	1.00
query18	0.22	0.20	0.20
query19	1.92	1.99	1.80
query20	0.01	0.02	0.01
query21	15.38	0.87	0.53
query22	0.74	1.19	0.63
query23	14.97	1.37	0.61
query24	7.53	1.08	1.52
query25	0.49	0.23	0.06
query26	0.56	0.16	0.13
query27	0.05	0.06	0.05
query28	9.75	0.79	0.44
query29	12.56	3.94	3.26
query30	0.26	0.09	0.06
query31	2.81	0.58	0.40
query32	3.23	0.53	0.47
query33	2.96	2.98	3.03
query34	15.60	5.10	4.46
query35	4.57	4.55	4.53
query36	0.66	0.50	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.14
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 105.64 s
Total hot run time: 30.81 s

swjtu-zhanglei · 2025-02-26T12:49:03Z

run buildall

doris-robot · 2025-02-26T13:07:41Z

TPC-H: Total hot run time: 31764 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cd1ff86b852a623e84fe8ae231f8498a50972929, data reload: false

------ Round 1 ----------------------------------
q1	17626	5377	5091	5091
q2	2047	300	169	169
q3	10415	1283	743	743
q4	10223	1008	535	535
q5	7535	2399	2344	2344
q6	197	168	132	132
q7	907	739	609	609
q8	9313	1269	1114	1114
q9	5032	4841	4657	4657
q10	6829	2319	1873	1873
q11	476	282	259	259
q12	350	351	222	222
q13	17756	3664	3116	3116
q14	234	223	214	214
q15	502	454	461	454
q16	636	619	597	597
q17	589	874	343	343
q18	6611	6317	6227	6227
q19	1211	953	569	569
q20	330	321	186	186
q21	3011	2298	2004	2004
q22	373	333	306	306
Total cold run time: 102203 ms
Total hot run time: 31764 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5156	5130	5132	5130
q2	240	326	237	237
q3	2167	2665	2294	2294
q4	1464	1812	1359	1359
q5	4229	4155	4198	4155
q6	207	162	122	122
q7	1868	1807	1668	1668
q8	2611	2589	2622	2589
q9	7375	7126	7087	7087
q10	2987	3265	2795	2795
q11	571	510	486	486
q12	668	736	620	620
q13	3602	3937	3232	3232
q14	275	290	271	271
q15	510	473	454	454
q16	634	670	657	657
q17	1168	1577	1366	1366
q18	7451	7308	7294	7294
q19	799	787	878	787
q20	1974	2025	1877	1877
q21	5429	5026	4888	4888
q22	629	606	575	575
Total cold run time: 52014 ms
Total hot run time: 49943 ms

doris-robot · 2025-02-26T13:19:06Z

TPC-DS: Total hot run time: 184582 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cd1ff86b852a623e84fe8ae231f8498a50972929, data reload: false

query1	982	392	383	383
query2	6549	1939	1861	1861
query3	6791	215	209	209
query4	26289	23755	23554	23554
query5	4282	672	494	494
query6	296	233	190	190
query7	4606	494	295	295
query8	307	244	240	240
query9	8594	2561	2552	2552
query10	448	310	268	268
query11	15198	15024	15236	15024
query12	152	105	103	103
query13	1650	504	376	376
query14	9622	6582	6808	6582
query15	211	190	179	179
query16	7162	648	498	498
query17	1227	712	565	565
query18	1951	392	312	312
query19	196	190	157	157
query20	125	115	129	115
query21	210	124	101	101
query22	4250	4190	4048	4048
query23	33722	33004	33057	33004
query24	7697	2392	2396	2392
query25	563	482	418	418
query26	1225	271	160	160
query27	2119	496	328	328
query28	3922	2432	2413	2413
query29	748	556	420	420
query30	232	196	160	160
query31	937	862	759	759
query32	77	63	63	63
query33	565	347	300	300
query34	790	849	494	494
query35	797	827	737	737
query36	972	976	867	867
query37	123	102	78	78
query38	4186	4096	4058	4058
query39	1456	1359	1379	1359
query40	208	114	100	100
query41	54	51	51	51
query42	121	102	105	102
query43	509	517	484	484
query44	1267	771	775	771
query45	180	168	161	161
query46	856	1039	642	642
query47	1753	1780	1699	1699
query48	369	407	295	295
query49	785	482	416	416
query50	669	717	421	421
query51	4195	4162	4141	4141
query52	103	112	93	93
query53	241	272	188	188
query54	497	485	400	400
query55	81	84	80	80
query56	260	255	254	254
query57	1126	1138	1061	1061
query58	245	236	243	236
query59	2581	2793	2500	2500
query60	282	257	254	254
query61	148	120	118	118
query62	795	758	674	674
query63	238	196	198	196
query64	4354	1040	649	649
query65	3242	3124	3156	3124
query66	1143	424	305	305
query67	15811	15466	15411	15411
query68	8141	861	494	494
query69	461	291	269	269
query70	1201	1159	1140	1140
query71	445	301	261	261
query72	5589	3518	3693	3518
query73	772	713	343	343
query74	8856	9363	8963	8963
query75	3747	3145	2706	2706
query76	3698	1170	756	756
query77	784	357	271	271
query78	9891	10114	9340	9340
query79	2610	805	590	590
query80	628	506	448	448
query81	495	270	242	242
query82	668	126	93	93
query83	208	181	154	154
query84	289	91	73	73
query85	785	350	385	350
query86	337	295	291	291
query87	4465	4363	4367	4363
query88	2941	2226	2215	2215
query89	388	323	280	280
query90	1918	189	192	189
query91	151	141	110	110
query92	78	59	60	59
query93	1186	1043	567	567
query94	675	409	295	295
query95	358	269	250	250
query96	487	544	271	271
query97	3350	3392	3272	3272
query98	226	209	205	205
query99	1437	1404	1262	1262
Total cold run time: 270607 ms
Total hot run time: 184582 ms

doris-robot · 2025-02-26T13:24:22Z

ClickBench: Total hot run time: 30.31 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cd1ff86b852a623e84fe8ae231f8498a50972929, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.03
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.56	0.54	0.55
query6	1.19	0.72	0.72
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.58	0.54	0.52
query10	0.58	0.56	0.57
query11	0.15	0.10	0.11
query12	0.14	0.11	0.12
query13	0.62	0.60	0.59
query14	2.67	2.72	2.85
query15	0.92	0.85	0.85
query16	0.39	0.37	0.37
query17	1.00	1.03	1.02
query18	0.21	0.19	0.20
query19	1.87	1.78	1.97
query20	0.01	0.01	0.01
query21	15.37	0.90	0.54
query22	0.75	1.25	0.74
query23	14.74	1.37	0.63
query24	7.07	1.86	0.37
query25	0.30	0.33	0.10
query26	0.71	0.15	0.14
query27	0.05	0.05	0.05
query28	8.78	0.85	0.44
query29	12.57	3.93	3.24
query30	0.25	0.09	0.07
query31	2.83	0.58	0.37
query32	3.21	0.53	0.47
query33	2.95	2.98	3.05
query34	15.82	5.15	4.46
query35	4.51	4.56	4.55
query36	0.70	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.03
query40	0.16	0.13	0.14
query41	0.09	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 104.05 s
Total hot run time: 30.31 s

swjtu-zhanglei · 2025-02-27T02:07:19Z

run feut

swjtu-zhanglei · 2025-03-04T08:20:48Z

run buildall

doris-robot · 2025-03-04T08:55:45Z

TPC-H: Total hot run time: 31644 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6e159b80665350dc168eb005cbbe98a5ff6ef83c, data reload: false

------ Round 1 ----------------------------------
q1	17598	5256	5071	5071
q2	2045	309	169	169
q3	11060	1219	735	735
q4	10245	1036	537	537
q5	7578	2410	2379	2379
q6	197	173	141	141
q7	903	746	629	629
q8	9323	1245	1060	1060
q9	4794	4686	4817	4686
q10	6856	2319	1906	1906
q11	478	278	264	264
q12	350	352	227	227
q13	17790	3705	3125	3125
q14	232	231	216	216
q15	524	448	468	448
q16	620	617	574	574
q17	596	848	344	344
q18	6570	6141	6242	6141
q19	1826	960	561	561
q20	324	310	188	188
q21	2808	2119	1942	1942
q22	371	334	301	301
Total cold run time: 103088 ms
Total hot run time: 31644 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5168	5514	5108	5108
q2	238	325	226	226
q3	2109	2703	2253	2253
q4	1445	1835	1383	1383
q5	4259	4105	4166	4105
q6	207	164	124	124
q7	1857	1815	1753	1753
q8	2627	2659	2602	2602
q9	7264	7156	7137	7137
q10	3023	3212	2760	2760
q11	607	519	491	491
q12	688	803	637	637
q13	3385	4001	3330	3330
q14	272	296	268	268
q15	502	475	480	475
q16	680	710	641	641
q17	1133	1622	1372	1372
q18	7775	7425	7304	7304
q19	844	821	920	821
q20	1965	2022	1872	1872
q21	5499	4974	4740	4740
q22	639	606	600	600
Total cold run time: 52186 ms
Total hot run time: 50002 ms

doris-robot · 2025-03-04T09:07:52Z

TPC-DS: Total hot run time: 190703 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6e159b80665350dc168eb005cbbe98a5ff6ef83c, data reload: false

query1	1336	974	927	927
query2	6242	1873	1884	1873
query3	11075	4513	4547	4513
query4	55981	25098	23041	23041
query5	4992	532	511	511
query6	339	202	191	191
query7	4896	505	299	299
query8	310	257	241	241
query9	5619	2560	2580	2560
query10	440	324	252	252
query11	14987	15182	14789	14789
query12	153	116	108	108
query13	1045	504	375	375
query14	10684	6350	7040	6350
query15	209	213	173	173
query16	7062	665	491	491
query17	1042	706	556	556
query18	1536	426	322	322
query19	200	192	165	165
query20	150	126	121	121
query21	212	127	104	104
query22	4590	4730	4549	4549
query23	34283	33298	33453	33298
query24	5703	2449	2422	2422
query25	452	466	395	395
query26	680	278	160	160
query27	1669	507	334	334
query28	2910	2489	2458	2458
query29	601	544	418	418
query30	220	209	160	160
query31	912	853	805	805
query32	73	62	62	62
query33	454	358	306	306
query34	769	876	494	494
query35	807	837	794	794
query36	956	988	921	921
query37	117	95	74	74
query38	4234	4252	4119	4119
query39	1496	1477	1410	1410
query40	211	126	112	112
query41	65	55	58	55
query42	126	107	112	107
query43	507	506	483	483
query44	1320	832	823	823
query45	187	180	169	169
query46	888	1105	657	657
query47	1849	1895	1798	1798
query48	385	428	304	304
query49	696	526	427	427
query50	707	749	411	411
query51	4301	4297	4241	4241
query52	112	105	104	104
query53	243	261	191	191
query54	480	497	413	413
query55	79	84	79	79
query56	264	282	268	268
query57	1166	1207	1169	1169
query58	249	261	242	242
query59	2705	2937	2699	2699
query60	297	287	261	261
query61	143	118	128	118
query62	749	756	678	678
query63	238	189	196	189
query64	1516	1045	706	706
query65	3252	3336	3267	3267
query66	730	413	312	312
query67	15892	15616	15364	15364
query68	5485	885	527	527
query69	519	315	275	275
query70	1237	1134	1146	1134
query71	443	305	275	275
query72	5933	3539	3676	3539
query73	1258	707	356	356
query74	9105	9144	8961	8961
query75	3294	3143	2745	2745
query76	3681	1181	742	742
query77	562	382	284	284
query78	10072	10230	9451	9451
query79	2804	819	588	588
query80	661	536	449	449
query81	502	266	233	233
query82	572	122	95	95
query83	177	173	147	147
query84	287	92	73	73
query85	772	351	305	305
query86	403	319	289	289
query87	4373	4502	4463	4463
query88	3555	2193	2201	2193
query89	411	319	285	285
query90	1851	189	194	189
query91	131	140	107	107
query92	74	59	56	56
query93	2282	1013	577	577
query94	675	413	315	315
query95	395	266	251	251
query96	491	562	272	272
query97	3367	3463	3284	3284
query98	225	204	204	204
query99	1331	1406	1251	1251
Total cold run time: 295721 ms
Total hot run time: 190703 ms

doris-robot · 2025-03-04T09:13:09Z

ClickBench: Total hot run time: 30.62 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6e159b80665350dc168eb005cbbe98a5ff6ef83c, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.61	0.10	0.10
query5	0.55	0.55	0.55
query6	1.19	0.72	0.71
query7	0.03	0.02	0.01
query8	0.04	0.04	0.04
query9	0.58	0.54	0.53
query10	0.57	0.58	0.57
query11	0.16	0.11	0.10
query12	0.15	0.11	0.12
query13	0.61	0.60	0.60
query14	2.81	2.68	2.69
query15	0.90	0.86	0.84
query16	0.37	0.37	0.40
query17	1.02	1.02	1.09
query18	0.22	0.20	0.20
query19	1.88	1.85	2.02
query20	0.01	0.01	0.01
query21	15.35	0.90	0.54
query22	0.74	1.23	0.70
query23	14.85	1.37	0.65
query24	7.89	1.33	0.55
query25	0.49	0.28	0.08
query26	0.57	0.16	0.14
query27	0.05	0.05	0.05
query28	8.78	0.85	0.43
query29	12.60	3.96	3.26
query30	0.24	0.09	0.07
query31	2.84	0.59	0.38
query32	3.23	0.53	0.46
query33	3.01	3.05	3.09
query34	15.80	5.14	4.50
query35	4.59	4.56	4.54
query36	0.66	0.49	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.21 s
Total hot run time: 30.62 s

github-actions · 2025-03-05T18:26:56Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-03-05T18:26:58Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsStorageVault.java

morningman · 2025-03-06T02:33:52Z

fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsStorageVault.java

+                                newProperties, "=", true, false, true, false));
+            }
+        } catch (IOException e) {
+            LOG.warn("checkConnectivity failed, properties:{}", new PrintableMap<>(


I think we'd better print vault name in log

fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsStorageVault.java

hust-hhb

LGTM

hust-hhb · 2025-03-06T03:48:42Z

If you just want to check the connection, you can do it this way. However, if you want to check whether the cluster is available, it's best to create a file, randomly write a string of characters, and then close the file, because if the datanode is down, creating a directory is ok in hdfs.

swjtu-zhanglei · 2025-03-06T08:18:45Z

run buildall

doris-robot · 2025-03-06T08:50:19Z

TPC-H: Total hot run time: 32745 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c5f111ba24ec9f08b6327d80467c2d70dc6faff7, data reload: false

------ Round 1 ----------------------------------
q1	17622	5250	5132	5132
q2	2043	297	169	169
q3	10497	1380	751	751
q4	10281	1045	542	542
q5	8197	2408	2391	2391
q6	198	168	135	135
q7	910	742	621	621
q8	9295	1337	1133	1133
q9	5126	4666	4859	4666
q10	6828	2324	1892	1892
q11	485	282	268	268
q12	348	353	212	212
q13	17767	3727	3132	3132
q14	226	250	234	234
q15	539	488	480	480
q16	616	618	585	585
q17	602	860	346	346
q18	6897	6431	6371	6371
q19	1268	953	539	539
q20	312	350	206	206
q21	2842	2278	1965	1965
q22	1078	1041	975	975
Total cold run time: 103977 ms
Total hot run time: 32745 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5238	5169	5211	5169
q2	244	329	235	235
q3	2172	2705	2298	2298
q4	1483	1886	1382	1382
q5	4308	4161	4190	4161
q6	212	165	124	124
q7	1999	1940	1778	1778
q8	2618	2584	2535	2535
q9	7197	7075	7270	7075
q10	3012	3237	2774	2774
q11	595	520	501	501
q12	699	784	604	604
q13	3527	3823	3250	3250
q14	285	301	273	273
q15	514	475	471	471
q16	649	697	647	647
q17	1175	1616	1312	1312
q18	7883	7599	7395	7395
q19	837	849	919	849
q20	2027	1986	1850	1850
q21	5504	5129	5013	5013
q22	1113	1049	1006	1006
Total cold run time: 53291 ms
Total hot run time: 50702 ms

doris-robot · 2025-03-06T09:01:51Z

TPC-DS: Total hot run time: 184721 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c5f111ba24ec9f08b6327d80467c2d70dc6faff7, data reload: false

query1	993	385	390	385
query2	6532	1892	1894	1892
query3	6789	217	217	217
query4	26752	23973	22975	22975
query5	4334	645	468	468
query6	283	191	188	188
query7	4603	505	293	293
query8	299	244	243	243
query9	8625	2547	2552	2547
query10	488	310	257	257
query11	15579	15079	15038	15038
query12	159	112	104	104
query13	1659	517	397	397
query14	8841	6294	6629	6294
query15	205	186	178	178
query16	7180	634	440	440
query17	1188	683	542	542
query18	1943	398	284	284
query19	183	192	150	150
query20	122	113	118	113
query21	237	123	103	103
query22	4288	4277	4118	4118
query23	33882	32983	33121	32983
query24	7706	2411	2377	2377
query25	529	456	397	397
query26	1226	267	156	156
query27	2513	485	329	329
query28	4291	2389	2394	2389
query29	754	547	422	422
query30	286	213	190	190
query31	923	859	773	773
query32	74	63	64	63
query33	561	358	300	300
query34	788	841	508	508
query35	794	816	746	746
query36	968	997	879	879
query37	121	99	77	77
query38	4189	4083	4323	4083
query39	1467	1401	1392	1392
query40	216	117	107	107
query41	61	56	60	56
query42	130	109	102	102
query43	501	528	477	477
query44	1270	786	805	786
query45	181	178	166	166
query46	851	1035	619	619
query47	1726	1764	1720	1720
query48	384	413	295	295
query49	806	529	435	435
query50	683	740	409	409
query51	4208	4133	4181	4133
query52	114	109	98	98
query53	229	261	192	192
query54	500	493	429	429
query55	92	86	84	84
query56	282	320	248	248
query57	1128	1119	1053	1053
query58	241	242	234	234
query59	2583	2656	2515	2515
query60	340	273	278	273
query61	117	115	120	115
query62	786	762	684	684
query63	234	185	192	185
query64	4391	990	645	645
query65	4460	4325	4301	4301
query66	1125	412	318	318
query67	15871	15611	15363	15363
query68	8223	895	504	504
query69	478	316	263	263
query70	1186	1135	1097	1097
query71	459	282	265	265
query72	5698	3566	3666	3566
query73	777	718	339	339
query74	8885	9097	8752	8752
query75	3763	3198	2682	2682
query76	3623	1176	739	739
query77	785	365	290	290
query78	10026	10247	9269	9269
query79	2717	830	588	588
query80	704	547	435	435
query81	468	248	221	221
query82	691	131	95	95
query83	208	177	150	150
query84	288	98	75	75
query85	793	344	307	307
query86	336	314	289	289
query87	4523	4443	4409	4409
query88	2806	2256	2238	2238
query89	425	321	285	285
query90	1928	210	210	210
query91	150	132	111	111
query92	77	63	56	56
query93	1682	1075	586	586
query94	660	391	278	278
query95	358	275	260	260
query96	488	550	272	272
query97	3379	3440	3257	3257
query98	232	218	203	203
query99	1436	1399	1298	1298
Total cold run time: 274116 ms
Total hot run time: 184721 ms

doris-robot · 2025-03-06T09:07:10Z

ClickBench: Total hot run time: 31.44 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c5f111ba24ec9f08b6327d80467c2d70dc6faff7, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.03
query3	0.23	0.07	0.07
query4	1.60	0.10	0.10
query5	0.56	0.55	0.55
query6	1.17	0.71	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.59	0.55	0.53
query10	0.57	0.57	0.57
query11	0.15	0.11	0.11
query12	0.14	0.11	0.12
query13	0.61	0.60	0.60
query14	2.67	2.85	2.81
query15	0.93	0.85	0.84
query16	0.39	0.38	0.37
query17	1.03	1.06	0.99
query18	0.21	0.20	0.19
query19	1.95	1.76	1.99
query20	0.01	0.01	0.01
query21	15.36	0.90	0.55
query22	0.75	1.10	0.72
query23	14.96	1.42	0.65
query24	6.84	1.57	1.30
query25	0.49	0.19	0.06
query26	0.62	0.16	0.13
query27	0.05	0.05	0.05
query28	9.95	0.81	0.42
query29	12.54	4.04	3.36
query30	0.24	0.08	0.07
query31	2.84	0.58	0.39
query32	3.22	0.54	0.46
query33	3.11	3.02	3.03
query34	15.98	5.14	4.53
query35	4.57	4.56	4.51
query36	0.68	0.49	0.50
query37	0.09	0.06	0.07
query38	0.04	0.04	0.04
query39	0.03	0.03	0.02
query40	0.16	0.12	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 105.64 s
Total hot run time: 31.44 s

dataroaring

LGTM

github-actions · 2025-03-07T02:26:57Z

PR approved by at least one committer and no changes requested.

…#48369) When creating an HDFS storage vault, perform connection and permission checks to avoid outputting error messages that could cause the storage vault to throw errors during usage after creation.

…torage vault #48369 (#48816) Cherry-picked from #48369 Co-authored-by: Lei Zhang <zhanglei@selectdb.com>

…apache#48369) When creating an HDFS storage vault, perform connection and permission checks to avoid outputting error messages that could cause the storage vault to throw errors during usage after creation.

swjtu-zhanglei requested review from dataroaring, gavinchou and w41ter as code owners February 26, 2025 08:12

swjtu-zhanglei force-pushed the master-fix11 branch from 07a9c0d to cd1ff86 Compare February 26, 2025 12:48

swjtu-zhanglei force-pushed the master-fix11 branch from cd1ff86 to 6e159b8 Compare March 4, 2025 08:20

gavinchou previously approved these changes Mar 5, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 5, 2025

github-actions bot added the reviewed label Mar 5, 2025

dataroaring previously approved these changes Mar 6, 2025

View reviewed changes

dataroaring added the dev/3.0.x label Mar 6, 2025

gavinchou changed the title ~~[feat](vault) Check hdfs connectivity when creating hdfs storage vault~~ [opt](vault) Check hdfs connectivity when creating hdfs storage vault Mar 6, 2025

morningman reviewed Mar 6, 2025

View reviewed changes

[feat](vault) Check hdfs connectivity when creating hdfs storage vault

0bd7045

swjtu-zhanglei added 2 commits March 6, 2025 11:23

Fix some comments

f60b91c

Fix some comments

22c87b8

swjtu-zhanglei dismissed stale reviews from dataroaring and gavinchou via 22c87b8 March 6, 2025 03:23

swjtu-zhanglei force-pushed the master-fix11 branch from 6e159b8 to 22c87b8 Compare March 6, 2025 03:23

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 6, 2025

hust-hhb approved these changes Mar 6, 2025

View reviewed changes

Merge branch 'master' into master-fix11

c5f111b

dataroaring approved these changes Mar 7, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 7, 2025

dataroaring merged commit 22cfc09 into apache:master Mar 7, 2025
24 checks passed

github-actions bot mentioned this pull request Mar 7, 2025

branch-3.0: [opt](vault) Check hdfs connectivity when creating hdfs storage vault #48369 #48816

Merged

dataroaring pushed a commit that referenced this pull request Mar 12, 2025

branch-3.0: [opt](vault) Check hdfs connectivity when creating hdfs s…

bf60457

…torage vault #48369 (#48816) Cherry-picked from #48369 Co-authored-by: Lei Zhang <zhanglei@selectdb.com>

dataroaring added dev/3.0.5-merged and removed dev/3.0.x labels Mar 12, 2025

gavinchou mentioned this pull request Apr 23, 2025

Release Notes 3.0.5 #50333

Closed

[opt](vault) Check hdfs connectivity when creating hdfs storage vault #48369

[opt](vault) Check hdfs connectivity when creating hdfs storage vault #48369

Uh oh!

Conversation

swjtu-zhanglei commented Feb 26, 2025 • edited by gavinchou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Feb 26, 2025

Uh oh!

swjtu-zhanglei commented Feb 26, 2025

Uh oh!

swjtu-zhanglei commented Feb 26, 2025

Uh oh!

doris-robot commented Feb 26, 2025

Uh oh!

doris-robot commented Feb 26, 2025

Uh oh!

doris-robot commented Feb 26, 2025

Uh oh!

swjtu-zhanglei commented Feb 26, 2025

Uh oh!

doris-robot commented Feb 26, 2025

Uh oh!

doris-robot commented Feb 26, 2025

Uh oh!

doris-robot commented Feb 26, 2025

Uh oh!

swjtu-zhanglei commented Feb 27, 2025

Uh oh!

swjtu-zhanglei commented Mar 4, 2025

Uh oh!

doris-robot commented Mar 4, 2025

Uh oh!

doris-robot commented Mar 4, 2025

Uh oh!

doris-robot commented Mar 4, 2025

Uh oh!

github-actions bot commented Mar 5, 2025

Uh oh!

github-actions bot commented Mar 5, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

morningman Mar 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hust-hhb left a comment

Choose a reason for hiding this comment

Uh oh!

hust-hhb commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

swjtu-zhanglei commented Mar 6, 2025

Uh oh!

doris-robot commented Mar 6, 2025

Uh oh!

doris-robot commented Mar 6, 2025

Uh oh!

doris-robot commented Mar 6, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 7, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

swjtu-zhanglei commented Feb 26, 2025 •

edited by gavinchou

Loading

hust-hhb commented Mar 6, 2025 •

edited

Loading