Skip to content

[feature](Vault) Support alter s3 storage vault using http interface #37537

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ByteYue wants to merge 3 commits into from
Closed

[feature](Vault) Support alter s3 storage vault using http interface #37537

ByteYue wants to merge 3 commits into from

Conversation

@ByteYue
Copy link
Contributor

@ByteYue ByteYue commented Jul 9, 2024

Proposed changes

Now user can alter theirs s3 storage vault's properties. Currently only ak and sk are supported.

@doris-robot
Copy link

doris-robot commented Jul 9, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

morrySnow
morrySnow requested changes Jul 9, 2024
View reviewed changes
@github-actions
Copy link
Contributor

github-actions bot commented Jul 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

2 similar comments
@github-actions
Copy link
Contributor

github-actions bot commented Jul 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

@github-actions
Copy link
Contributor

github-actions bot commented Jul 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

@ByteYue ByteYue force-pushed the add_alter_storage_vault branch from 940247c to d108f91 Compare July 10, 2024 07:15
@ByteYue ByteYue changed the title [feature](Vault) Support alter s3 storage vault stmt [feature](Vault) Support alter s3 storage vault using http interface Jul 10, 2024
@ByteYue
Copy link
Contributor Author

ByteYue commented Jul 10, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

TPC-H: Total hot run time: 42663 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9374c1c40503cc9dfcb19a2c84c70a3b86b07ddc, data reload: false

------ Round 1 ----------------------------------
q1	18497	4917	5055	4917
q2	2041	196	186	186
q3	10537	1219	1170	1170
q4	10228	925	811	811
q5	7536	2880	2890	2880
q6	245	139	134	134
q7	1053	606	615	606
q8	9232	2295	2314	2295
q9	9151	6948	6935	6935
q10	9068	3848	3842	3842
q11	469	239	244	239
q12	455	223	219	219
q13	17815	2981	2987	2981
q14	283	223	219	219
q15	542	467	486	467
q16	517	369	376	369
q17	1038	702	738	702
q18	8157	7590	7512	7512
q19	7087	1573	1589	1573
q20	699	319	321	319
q21	5138	3944	4137	3944
q22	412	344	343	343
Total cold run time: 120200 ms
Total hot run time: 42663 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4857	4686	4878	4686
q2	430	282	281	281
q3	3341	3000	3063	3000
q4	2091	1722	1722	1722
q5	5650	5698	5708	5698
q6	251	137	132	132
q7	2352	1896	1880	1880
q8	3532	3741	3748	3741
q9	8885	8910	8790	8790
q10	4214	3955	4001	3955
q11	641	479	490	479
q12	857	664	626	626
q13	15899	3148	3159	3148
q14	319	284	263	263
q15	546	485	479	479
q16	501	448	423	423
q17	2011	1663	1580	1580
q18	8318	8048	7985	7985
q19	2031	1927	1732	1732
q20	2730	1866	1866	1866
q21	5025	5207	5137	5137
q22	643	557	570	557
Total cold run time: 75124 ms
Total hot run time: 58160 ms

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

TPC-DS: Total hot run time: 175032 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9374c1c40503cc9dfcb19a2c84c70a3b86b07ddc, data reload: false

query1	919	375	366	366
query2	6357	2446	2456	2446
query3	6651	207	213	207
query4	27412	17495	17386	17386
query5	3658	488	482	482
query6	287	162	167	162
query7	4584	302	286	286
query8	311	301	301	301
query9	8605	2398	2357	2357
query10	455	282	265	265
query11	11742	10180	10313	10180
query12	111	87	84	84
query13	1640	372	370	370
query14	10321	7594	7883	7594
query15	250	189	186	186
query16	7798	313	299	299
query17	1376	537	521	521
query18	1908	276	273	273
query19	208	157	153	153
query20	92	82	82	82
query21	208	127	135	127
query22	4332	4320	4056	4056
query23	34051	33691	33772	33691
query24	11081	2848	2841	2841
query25	643	431	409	409
query26	704	153	151	151
query27	2274	290	280	280
query28	5671	2161	2111	2111
query29	936	650	669	650
query30	256	160	158	158
query31	1003	759	781	759
query32	101	62	59	59
query33	691	327	305	305
query34	959	495	505	495
query35	685	602	608	602
query36	1147	996	960	960
query37	140	77	86	77
query38	2843	2786	2726	2726
query39	868	815	813	813
query40	205	119	117	117
query41	55	52	50	50
query42	121	99	101	99
query43	590	551	562	551
query44	1080	739	744	739
query45	189	163	160	160
query46	1057	696	733	696
query47	1886	1768	1782	1768
query48	385	293	292	292
query49	831	403	410	403
query50	776	397	395	395
query51	6851	6814	6732	6732
query52	112	100	90	90
query53	355	285	288	285
query54	877	450	446	446
query55	78	74	74	74
query56	303	263	273	263
query57	1150	1082	1013	1013
query58	234	258	252	252
query59	3399	3394	3145	3145
query60	295	274	279	274
query61	97	93	92	92
query62	767	607	645	607
query63	323	286	285	285
query64	9149	2151	1622	1622
query65	3173	3113	3116	3113
query66	698	327	330	327
query67	15846	15148	15047	15047
query68	5490	552	539	539
query69	679	419	362	362
query70	1207	1088	1153	1088
query71	453	277	319	277
query72	8753	6043	5829	5829
query73	820	326	327	326
query74	5838	5521	5484	5484
query75	3833	2694	2675	2675
query76	3334	962	924	924
query77	664	301	303	301
query78	9516	8937	8897	8897
query79	3385	516	509	509
query80	1909	474	465	465
query81	593	217	214	214
query82	1470	136	140	136
query83	354	170	172	170
query84	273	90	88	88
query85	1350	328	302	302
query86	445	314	276	276
query87	3284	3107	3070	3070
query88	4464	2442	2481	2442
query89	547	387	379	379
query90	1791	192	190	190
query91	134	106	107	106
query92	63	50	47	47
query93	4885	518	514	514
query94	993	214	205	205
query95	413	312	325	312
query96	599	280	269	269
query97	3187	3044	2989	2989
query98	240	203	202	202
query99	1696	1270	1238	1238
Total cold run time: 286822 ms
Total hot run time: 175032 ms

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

ClickBench: Total hot run time: 30.89 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9374c1c40503cc9dfcb19a2c84c70a3b86b07ddc, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.70	0.07	0.07
query5	0.50	0.50	0.48
query6	1.13	0.72	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.50
query10	0.54	0.55	0.56
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.58	0.58	0.58
query14	0.77	0.75	0.82
query15	0.83	0.83	0.81
query16	0.37	0.36	0.38
query17	1.04	1.04	0.99
query18	0.23	0.22	0.21
query19	1.89	1.76	1.70
query20	0.01	0.02	0.01
query21	15.40	0.79	0.67
query22	4.13	6.80	2.21
query23	18.36	1.40	1.35
query24	2.09	0.23	0.23
query25	0.15	0.08	0.08
query26	0.30	0.20	0.20
query27	0.45	0.23	0.22
query28	13.23	1.01	0.99
query29	12.63	3.36	3.34
query30	0.25	0.06	0.06
query31	2.87	0.39	0.39
query32	3.29	0.48	0.46
query33	2.90	2.88	2.86
query34	16.89	4.32	4.30
query35	4.41	4.36	4.49
query36	0.64	0.47	0.48
query37	0.18	0.15	0.16
query38	0.16	0.16	0.15
query39	0.04	0.03	0.04
query40	0.15	0.13	0.12
query41	0.09	0.06	0.05
query42	0.05	0.04	0.05
query43	0.04	0.03	0.04
Total cold run time: 109.55 s
Total hot run time: 30.89 s

@morrySnow morrySnow self-requested a review July 10, 2024 10:55
@ByteYue ByteYue force-pushed the add_alter_storage_vault branch from 9374c1c to 663b1d2 Compare July 10, 2024 11:13
@ByteYue
Copy link
Contributor Author

ByteYue commented Jul 10, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jul 10, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

TPC-H: Total hot run time: 40308 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 663b1d2a7c4767ad1e0c48e69fee1c1bf83bcf7d, data reload: false

------ Round 1 ----------------------------------
q1	18466	5232	4437	4437
q2	2602	199	201	199
q3	11219	1159	1115	1115
q4	10577	834	823	823
q5	7544	2772	2703	2703
q6	229	141	141	141
q7	996	622	630	622
q8	9277	2141	2176	2141
q9	8923	6546	6565	6546
q10	8955	3720	3731	3720
q11	467	242	247	242
q12	452	237	233	233
q13	18874	2984	3008	2984
q14	268	231	215	215
q15	535	486	478	478
q16	499	381	377	377
q17	996	615	716	615
q18	8094	7584	7372	7372
q19	5333	1450	1414	1414
q20	653	327	329	327
q21	5119	3256	4053	3256
q22	404	348	349	348
Total cold run time: 120482 ms
Total hot run time: 40308 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4413	4309	4350	4309
q2	383	274	275	274
q3	2990	2792	2683	2683
q4	1842	1578	1628	1578
q5	5325	5321	5321	5321
q6	224	135	134	134
q7	2126	1771	1728	1728
q8	3221	3419	3353	3353
q9	8408	8406	8387	8387
q10	3882	3676	3703	3676
q11	570	486	518	486
q12	774	580	624	580
q13	17719	2972	3002	2972
q14	306	266	276	266
q15	524	485	484	484
q16	463	408	420	408
q17	1831	1474	1475	1474
q18	7771	7761	7536	7536
q19	1722	1512	1459	1459
q20	2024	1795	1781	1781
q21	4756	4836	4806	4806
q22	637	546	542	542
Total cold run time: 71911 ms
Total hot run time: 54237 ms

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

TPC-DS: Total hot run time: 174169 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 663b1d2a7c4767ad1e0c48e69fee1c1bf83bcf7d, data reload: false

query1	916	385	366	366
query2	6446	2560	2367	2367
query3	6651	211	220	211
query4	26651	17450	17150	17150
query5	4180	506	493	493
query6	288	175	159	159
query7	4597	295	291	291
query8	331	292	300	292
query9	8499	2403	2375	2375
query10	455	288	269	269
query11	11544	10003	10019	10003
query12	135	85	79	79
query13	1640	375	368	368
query14	9469	7594	7829	7594
query15	248	177	191	177
query16	7749	312	307	307
query17	1375	533	525	525
query18	1913	284	300	284
query19	193	151	151	151
query20	89	83	82	82
query21	208	128	125	125
query22	4587	4134	3944	3944
query23	33956	33144	33089	33089
query24	12278	2846	2826	2826
query25	680	413	376	376
query26	1797	161	165	161
query27	2952	274	272	272
query28	7758	2066	2058	2058
query29	1128	641	624	624
query30	291	147	146	146
query31	947	715	731	715
query32	94	54	56	54
query33	774	295	292	292
query34	965	494	489	489
query35	675	607	576	576
query36	1107	992	939	939
query37	204	81	79	79
query38	2838	2779	2765	2765
query39	844	792	819	792
query40	284	124	123	123
query41	53	52	53	52
query42	117	100	103	100
query43	607	545	554	545
query44	1251	758	735	735
query45	195	165	159	159
query46	1087	734	719	719
query47	1857	1740	1759	1740
query48	377	309	304	304
query49	1175	430	422	422
query50	798	408	410	408
query51	6902	6858	6780	6780
query52	116	93	99	93
query53	365	302	299	299
query54	977	460	452	452
query55	77	78	75	75
query56	335	290	280	280
query57	1116	1062	1040	1040
query58	262	239	265	239
query59	3519	3266	3309	3266
query60	306	273	273	273
query61	96	92	98	92
query62	812	649	660	649
query63	326	302	297	297
query64	10532	2317	7508	2317
query65	3182	3137	3096	3096
query66	1358	350	334	334
query67	15501	14970	15052	14970
query68	4558	555	547	547
query69	481	340	334	334
query70	1160	1085	1156	1085
query71	397	284	311	284
query72	7120	5463	5323	5323
query73	743	332	323	323
query74	5949	5476	5447	5447
query75	3395	2668	2704	2668
query76	2465	957	1060	957
query77	465	318	310	310
query78	9406	9586	8808	8808
query79	2096	531	523	523
query80	2348	557	484	484
query81	579	227	226	226
query82	724	149	150	149
query83	291	174	172	172
query84	263	92	90	90
query85	2144	325	309	309
query86	488	313	317	313
query87	3239	3093	3151	3093
query88	3547	2466	2449	2449
query89	500	413	395	395
query90	1892	194	195	194
query91	131	104	105	104
query92	63	49	51	49
query93	2303	525	525	525
query94	1289	218	216	216
query95	411	319	315	315
query96	630	284	267	267
query97	3204	3023	3042	3023
query98	222	206	195	195
query99	1588	1268	1270	1268
Total cold run time: 285126 ms
Total hot run time: 174169 ms

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

ClickBench: Total hot run time: 30.68 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 663b1d2a7c4767ad1e0c48e69fee1c1bf83bcf7d, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.03	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.49	0.49
query6	1.14	0.72	0.73
query7	0.02	0.02	0.02
query8	0.06	0.04	0.04
query9	0.53	0.49	0.48
query10	0.53	0.53	0.55
query11	0.15	0.12	0.12
query12	0.14	0.12	0.12
query13	0.60	0.58	0.59
query14	0.77	0.78	0.78
query15	0.84	0.82	0.81
query16	0.38	0.35	0.37
query17	1.03	0.97	1.00
query18	0.22	0.21	0.22
query19	1.78	1.69	1.72
query20	0.01	0.01	0.01
query21	15.41	0.78	0.67
query22	4.07	7.27	1.92
query23	18.30	1.41	1.37
query24	2.14	0.21	0.21
query25	0.16	0.09	0.08
query26	0.30	0.20	0.22
query27	0.45	0.24	0.23
query28	13.34	1.02	1.00
query29	12.61	3.32	3.31
query30	0.25	0.06	0.05
query31	2.89	0.38	0.39
query32	3.26	0.46	0.47
query33	2.87	2.88	2.86
query34	16.80	4.34	4.34
query35	4.43	4.49	4.45
query36	0.65	0.46	0.46
query37	0.18	0.16	0.15
query38	0.15	0.15	0.15
query39	0.05	0.04	0.04
query40	0.15	0.12	0.13
query41	0.09	0.05	0.05
query42	0.06	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.39 s
Total hot run time: 30.68 s

gavinchou
gavinchou requested changes Jul 13, 2024
View reviewed changes
cloud/src/meta-service/meta_service_resource.cpp
LOG(INFO) << vault_msg;
}

static int alter_storage_vault(InstanceInfoPB& instance, std::unique_ptr<Transaction> txn,
Copy link
Contributor

@gavinchou gavinchou Jul 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add an UT to test it.
And refer to MetaServiceImpl::alter_obj_store_info and MetaServiceImpl::alter_instance to abstract the behavior and make as flexible as possible.

Copy link
Contributor Author

@ByteYue ByteYue Jul 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ut and regression cases would be involved in #37606.

@ByteYue
Copy link
Contributor Author

ByteYue commented Jul 14, 2024

Included in #37606

@ByteYue ByteYue closed this Jul 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou requested changes

@platoneko platoneko platoneko approved these changes

@morrySnow morrySnow Awaiting requested review from morrySnow

Assignees

No one assigned

Labels

meta-change

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ByteYue @doris-robot @gavinchou @platoneko @morrySnow