[feat](catalog) Support passing credentials_provider_type to BE for S3 access #59082

zy-kkk · 2025-12-16T08:38:13Z

#58740

This is the backend change for PR #58740, with additional test cases added on top of it.

…3 access

Thearas · 2025-12-16T08:38:18Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

CalvinKirs · 2025-12-16T10:50:17Z

run buildall

CalvinKirs · 2025-12-16T11:11:18Z

run buildall

CalvinKirs · 2025-12-16T11:20:03Z

run buildall

doris-robot · 2025-12-16T11:41:04Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.82% (1752/2195)
Line Coverage	65.69% (30740/46794)
Region Coverage	66.45% (15366/23123)
Branch Coverage	56.73% (8159/14382)

hello-stephen · 2025-12-16T12:34:48Z

FE UT Coverage Report

Increment line coverage 85.71% (6/7) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2025-12-16T15:50:08Z

FE Regression Coverage Report

Increment line coverage 42.86% (3/7) 🎉
Increment coverage report
Complete coverage report

CalvinKirs · 2025-12-17T06:21:34Z

run buildall

...src/main/java/org/apache/doris/datasource/property/common/AwsCredentialsProviderFactory.java

doris-robot · 2025-12-17T06:53:37Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.82% (1752/2195)
Line Coverage	65.72% (30754/46794)
Region Coverage	66.48% (15372/23123)
Branch Coverage	56.76% (8163/14382)

CalvinKirs · 2025-12-17T07:26:17Z

run buildall

doris-robot · 2025-12-17T07:49:13Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.82% (1752/2195)
Line Coverage	65.75% (30765/46794)
Region Coverage	66.42% (15359/23123)
Branch Coverage	56.77% (8165/14382)

hello-stephen · 2025-12-17T08:42:10Z

FE UT Coverage Report

Increment line coverage 85.71% (6/7) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-12-17T09:13:33Z

TPC-H: Total hot run time: 36502 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b68c9f67fa528884eea7f94d368a07b0426c15ae, data reload: false

------ Round 1 ----------------------------------
q1	17659	4248	4062	4062
q2	2071	362	241	241
q3	10122	1330	764	764
q4	10234	850	312	312
q5	7529	2179	1971	1971
q6	194	167	138	138
q7	1042	851	727	727
q8	9361	1482	1133	1133
q9	7098	5375	5405	5375
q10	6868	2384	1960	1960
q11	530	329	308	308
q12	699	728	581	581
q13	17801	3668	3058	3058
q14	284	292	278	278
q15	599	541	527	527
q16	698	734	624	624
q17	692	823	527	527
q18	7817	7695	8011	7695
q19	1163	1005	672	672
q20	434	377	275	275
q21	4579	4317	4286	4286
q22	1162	1095	988	988
Total cold run time: 108636 ms
Total hot run time: 36502 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4425	4227	4294	4227
q2	327	413	337	337
q3	2487	2967	2548	2548
q4	1431	1825	1373	1373
q5	4557	4411	4486	4411
q6	212	166	126	126
q7	2055	1946	1835	1835
q8	2754	2563	2490	2490
q9	7535	7361	6978	6978
q10	2907	3100	2697	2697
q11	580	496	486	486
q12	627	716	554	554
q13	3281	3693	3083	3083
q14	264	284	273	273
q15	540	494	492	492
q16	612	654	613	613
q17	1113	1572	1337	1337
q18	7243	7067	7053	7053
q19	843	783	830	783
q20	1885	1970	1824	1824
q21	4664	4191	4224	4191
q22	1103	1041	964	964
Total cold run time: 51445 ms
Total hot run time: 48675 ms

doris-robot · 2025-12-17T09:24:32Z

TPC-DS: Total hot run time: 178900 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b68c9f67fa528884eea7f94d368a07b0426c15ae, data reload: false

query5	4765	594	446	446
query6	340	229	210	210
query7	4245	490	280	280
query8	315	255	256	255
query9	8809	2511	2552	2511
query10	522	372	340	340
query11	15117	14855	14795	14795
query12	189	117	121	117
query13	1283	505	406	406
query14	5964	3038	2741	2741
query14_1	2597	2597	2610	2597
query15	210	195	176	176
query16	926	423	462	423
query17	1096	705	568	568
query18	2424	453	341	341
query19	218	234	202	202
query20	123	115	111	111
query21	216	140	113	113
query22	3919	4086	4083	4083
query23	16608	16325	16144	16144
query23_1	15944	16010	16061	16010
query24	7386	1638	1198	1198
query24_1	1242	1232	1266	1232
query25	604	508	462	462
query26	1263	277	174	174
query27	2739	456	315	315
query28	4517	2138	2124	2124
query29	841	592	476	476
query30	317	252	222	222
query31	806	709	610	610
query32	87	72	73	72
query33	587	334	294	294
query34	890	881	546	546
query35	769	802	729	729
query36	840	907	830	830
query37	125	95	75	75
query38	2872	2904	2830	2830
query39	761	730	709	709
query39_1	707	697	723	697
query40	225	137	120	120
query41	71	64	63	63
query42	113	107	109	107
query43	443	434	405	405
query44	1314	736	740	736
query45	196	192	185	185
query46	875	977	623	623
query47	1664	1702	1609	1609
query48	332	330	259	259
query49	614	431	348	348
query50	660	281	233	233
query51	3873	3839	3764	3764
query52	104	111	97	97
query53	313	351	297	297
query54	285	254	249	249
query55	81	78	78	78
query56	295	294	287	287
query57	1119	1134	1074	1074
query58	274	255	252	252
query59	2390	2497	2347	2347
query60	308	308	291	291
query61	158	158	161	158
query62	726	659	621	621
query63	328	291	312	291
query64	5052	1301	1005	1005
query65	4051	3940	3963	3940
query66	1470	427	323	323
query67	15423	14960	14979	14960
query68	2775	1038	785	785
query69	496	361	306	306
query70	1022	1007	959	959
query71	328	306	282	282
query72	6241	5078	5136	5078
query73	549	647	309	309
query74	8604	8750	8572	8572
query75	3119	3160	2801	2801
query76	3454	1126	729	729
query77	514	400	298	298
query78	9405	9570	8852	8852
query79	1023	864	605	605
query80	699	661	554	554
query81	493	268	235	235
query82	216	134	131	131
query83	266	257	244	244
query84	260	116	103	103
query85	861	512	454	454
query86	301	299	273	273
query87	3019	3055	2969	2969
query88	3229	2291	2276	2276
query89	476	418	389	389
query90	2084	159	157	157
query91	175	168	148	148
query92	69	72	63	63
query93	969	927	562	562
query94	364	309	306	306
query95	574	374	310	310
query96	600	451	209	209
query97	2296	2352	2247	2247
query98	212	190	199	190
query99	1268	1293	1217	1217
Total cold run time: 251930 ms
Total hot run time: 178900 ms

doris-robot · 2025-12-17T09:29:35Z

ClickBench: Total hot run time: 27.26 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b68c9f67fa528884eea7f94d368a07b0426c15ae, data reload: false

query1	0.06	0.05	0.05
query2	0.11	0.06	0.05
query3	0.26	0.08	0.08
query4	1.60	0.11	0.10
query5	0.29	0.26	0.26
query6	1.16	0.65	0.64
query7	0.03	0.03	0.03
query8	0.05	0.04	0.05
query9	0.58	0.51	0.52
query10	0.56	0.56	0.56
query11	0.16	0.11	0.10
query12	0.15	0.11	0.12
query13	0.62	0.60	0.60
query14	0.98	0.98	0.99
query15	0.80	0.81	0.81
query16	0.42	0.40	0.39
query17	1.08	1.08	1.05
query18	0.23	0.22	0.21
query19	1.89	1.79	1.80
query20	0.02	0.01	0.01
query21	15.43	0.31	0.15
query22	4.78	0.06	0.05
query23	16.09	0.30	0.11
query24	0.96	0.76	0.19
query25	0.10	0.09	0.09
query26	0.14	0.14	0.13
query27	0.07	0.05	0.05
query28	4.15	1.22	1.03
query29	12.60	4.06	3.19
query30	0.27	0.14	0.11
query31	2.82	0.63	0.38
query32	3.24	0.57	0.48
query33	2.97	3.06	3.02
query34	17.08	5.13	4.52
query35	4.59	4.58	4.59
query36	0.67	0.49	0.48
query37	0.10	0.06	0.07
query38	0.07	0.04	0.04
query39	0.04	0.02	0.02
query40	0.18	0.15	0.13
query41	0.09	0.03	0.02
query42	0.05	0.03	0.02
query43	0.04	0.04	0.04
Total cold run time: 97.58 s
Total hot run time: 27.26 s

hello-stephen · 2025-12-17T13:07:25Z

FE Regression Coverage Report

Increment line coverage 85.71% (6/7) 🎉
Increment coverage report
Complete coverage report

CalvinKirs · 2025-12-17T13:48:52Z

run feut

CalvinKirs · 2025-12-17T13:48:58Z

run beut

zy-kkk · 2025-12-18T07:13:14Z

run buildall

CalvinKirs · 2025-12-18T09:52:37Z

run vault_p0

github-actions · 2025-12-18T09:53:56Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-12-18T09:53:59Z

PR approved by anyone and no changes requested.

doris-robot · 2025-12-18T10:14:57Z

BE UT Coverage Report

Increment line coverage 48.15% (13/27) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.36% (18817/35266)
Line Coverage	39.20% (174438/445043)
Region Coverage	33.72% (134858/399888)
Branch Coverage	34.67% (58166/167778)

hello-stephen · 2025-12-18T11:27:50Z

BE Regression && UT Coverage Report

Increment line coverage 66.67% (18/27) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	72.26% (24971/34559)
Line Coverage	59.03% (262320/444392)
Region Coverage	53.93% (218105/404454)
Branch Coverage	55.45% (93462/168564)

hello-stephen · 2025-12-18T11:38:12Z

FE Regression Coverage Report

Increment line coverage 85.71% (6/7) 🎉
Increment coverage report
Complete coverage report

…3 access (#59082) #58740 This is the backend change for PR #58740, with additional test cases added on top of it. --------- Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>

… to BE for S3 access #59082 (#59158) Cherry-picked from #59082 Co-authored-by: zy-kkk <zhongyk10@gmail.com> Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>

…3 access (#59082) #58740 This is the backend change for PR #58740, with additional test cases added on top of it. --------- Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>

… to BE for S3 access #59082 (#59246) Cherry-picked from #59082 Co-authored-by: zy-kkk <zhongyk10@gmail.com> Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>

[feat](catalog) Support passing credentials_provider_type to BE for S…

3c71435

…3 access

test

46dd48b

test

8d31420

test

0734644

test

ab87c97

CalvinKirs marked this pull request as ready for review December 17, 2025 06:21

morningman reviewed Dec 17, 2025

View reviewed changes

...src/main/java/org/apache/doris/datasource/property/common/AwsCredentialsProviderFactory.java Show resolved Hide resolved

fix test

b68c9f6

CalvinKirs added dev/3.1.x dev/4.0.x labels Dec 17, 2025

fix ut

fa74d0e

CalvinKirs approved these changes Dec 18, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 18, 2025

github-actions bot added the reviewed label Dec 18, 2025

morningman approved these changes Dec 18, 2025

View reviewed changes

morningman merged commit 86f0b3a into apache:master Dec 18, 2025
27 of 29 checks passed

github-actions bot mentioned this pull request Dec 18, 2025

branch-3.1: [feat](catalog) Support passing credentials_provider_type to BE for S3 access #59082 #59158

Merged

github-actions bot added the dev/4.0.x-conflict label Dec 18, 2025

zy-kkk deleted the aws-instance-profile-be branch December 19, 2025 03:24

morningman added dev/3.1.4-merged and removed dev/3.1.x labels Dec 19, 2025

CalvinKirs added dev/4.0.x and removed dev/4.0.x dev/4.0.x-conflict labels Dec 22, 2025

github-actions bot added the dev/4.0.x-conflict label Dec 22, 2025

zy-kkk added dev/4.0.x and removed dev/4.0.x dev/4.0.x-conflict labels Dec 22, 2025

github-actions bot mentioned this pull request Dec 22, 2025

branch-4.0: [feat](catalog) Support passing credentials_provider_type to BE for S3 access #59082 #59246

Merged

yiguolei added dev/4.0.3-merged and removed dev/4.0.x labels Dec 23, 2025

[feat](catalog) Support passing credentials_provider_type to BE for S3 access #59082

[feat](catalog) Support passing credentials_provider_type to BE for S3 access #59082

Uh oh!

Conversation

zy-kkk commented Dec 16, 2025 • edited by CalvinKirs Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Thearas commented Dec 16, 2025

Uh oh!

CalvinKirs commented Dec 16, 2025

Uh oh!

CalvinKirs commented Dec 16, 2025

Uh oh!

CalvinKirs commented Dec 16, 2025

Uh oh!

doris-robot commented Dec 16, 2025

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Dec 16, 2025

FE UT Coverage Report

Uh oh!

hello-stephen commented Dec 16, 2025

FE Regression Coverage Report

Uh oh!

CalvinKirs commented Dec 17, 2025

Uh oh!

Uh oh!

doris-robot commented Dec 17, 2025

Cloud UT Coverage Report

Uh oh!

CalvinKirs commented Dec 17, 2025

Uh oh!

doris-robot commented Dec 17, 2025

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Dec 17, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Dec 17, 2025

Uh oh!

doris-robot commented Dec 17, 2025

Uh oh!

doris-robot commented Dec 17, 2025

Uh oh!

hello-stephen commented Dec 17, 2025

FE Regression Coverage Report

Uh oh!

CalvinKirs commented Dec 17, 2025

Uh oh!

CalvinKirs commented Dec 17, 2025

Uh oh!

zy-kkk commented Dec 18, 2025

Uh oh!

CalvinKirs commented Dec 18, 2025

Uh oh!

github-actions bot commented Dec 18, 2025

Uh oh!

github-actions bot commented Dec 18, 2025

Uh oh!

doris-robot commented Dec 18, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Dec 18, 2025

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Dec 18, 2025

FE Regression Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

zy-kkk commented Dec 16, 2025 •

edited by CalvinKirs

Loading