Skip to content

[opt](vault) Do not use latest_fs() in vault mode #40516

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
gavinchou merged 1 commit into from
Sep 10, 2024
Merged

[opt](vault) Do not use latest_fs() in vault mode #40516

gavinchou merged 1 commit into from
Sep 10, 2024

Conversation

@gavinchou
Copy link
Contributor

@gavinchou gavinchou commented Sep 8, 2024

To prevent incorrect storage backends selected by loading data, e.g. empty vault id passed from FE, we should not use latest_fs() in vault mode.

@doris-robot
Copy link

doris-robot commented Sep 8, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@gavinchou
Copy link
Contributor Author

gavinchou commented Sep 8, 2024

run buildall

@gavinchou
[opt](vault) Do not use latest_fs() for vault mode
64d7fc4 
To prevent incorrect storage backends selected by loading data,
e.g. empty vault id passed from FE, we should not use latest_fs() in vault mode.
@gavinchou gavinchou force-pushed the gavin-disable-vault-latest-fs branch from a9b77d1 to 64d7fc4 Compare September 8, 2024 15:58
@gavinchou
Copy link
Contributor Author

gavinchou commented Sep 8, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Sep 8, 2024

TeamCity be ut coverage result:
Function Coverage: 36.83% (9382/25474)
Line Coverage: 28.23% (77383/274105)
Region Coverage: 27.63% (39960/144609)
Branch Coverage: 24.27% (20336/83802)
Coverage Report: http://coverage.selectdb-in.cc/coverage/64d7fc4ed979eda13ea87e609f407a375dc89ad4_64d7fc4ed979eda13ea87e609f407a375dc89ad4/report/index.html

@doris-robot
Copy link

doris-robot commented Sep 8, 2024

TPC-H: Total hot run time: 38404 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 64d7fc4ed979eda13ea87e609f407a375dc89ad4, data reload: false

------ Round 1 ----------------------------------
q1	17947	5271	4350	4350
q2	2031	193	209	193
q3	11445	968	1102	968
q4	10465	692	718	692
q5	7765	2897	2862	2862
q6	226	139	139	139
q7	968	619	620	619
q8	9319	2091	2081	2081
q9	7370	6539	6614	6539
q10	6994	2231	2252	2231
q11	463	240	235	235
q12	401	227	219	219
q13	17773	3089	3130	3089
q14	280	234	242	234
q15	545	489	480	480
q16	542	435	429	429
q17	986	724	660	660
q18	7571	6906	6920	6906
q19	1395	1106	1133	1106
q20	705	339	323	323
q21	4187	3149	3033	3033
q22	1125	1026	1016	1016
Total cold run time: 110503 ms
Total hot run time: 38404 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4372	4506	4283	4283
q2	376	286	288	286
q3	2875	2716	2642	2642
q4	1962	1637	1705	1637
q5	5713	5702	5781	5702
q6	233	134	139	134
q7	2241	1903	1853	1853
q8	3300	3498	3520	3498
q9	8868	8940	8826	8826
q10	3618	3410	3369	3369
q11	630	498	510	498
q12	830	669	669	669
q13	14933	3132	3313	3132
q14	318	278	286	278
q15	545	492	505	492
q16	554	514	514	514
q17	1843	1550	1535	1535
q18	8253	7847	7937	7847
q19	1779	1764	1624	1624
q20	2179	1928	1883	1883
q21	5865	5565	5419	5419
q22	1121	1059	1060	1059
Total cold run time: 72408 ms
Total hot run time: 57180 ms

@doris-robot
Copy link

doris-robot commented Sep 8, 2024

TPC-DS: Total hot run time: 192514 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 64d7fc4ed979eda13ea87e609f407a375dc89ad4, data reload: false

query1	1255	907	884	884
query2	6355	1883	1920	1883
query3	10597	4000	4134	4000
query4	59551	26994	23265	23265
query5	5346	486	490	486
query6	405	159	170	159
query7	5777	295	285	285
query8	332	230	229	229
query9	9017	2502	2483	2483
query10	503	292	261	261
query11	18195	15190	15394	15190
query12	167	114	100	100
query13	1584	396	394	394
query14	11132	7349	7348	7348
query15	223	178	164	164
query16	7555	490	454	454
query17	1095	560	567	560
query18	2066	295	289	289
query19	288	149	147	147
query20	119	118	126	118
query21	206	104	103	103
query22	4777	4761	4577	4577
query23	34721	33256	33655	33256
query24	5981	2839	2903	2839
query25	524	368	398	368
query26	683	152	156	152
query27	1798	272	271	271
query28	3900	2041	2030	2030
query29	694	401	400	400
query30	233	151	155	151
query31	922	752	766	752
query32	84	54	54	54
query33	473	310	317	310
query34	865	466	463	463
query35	849	705	731	705
query36	1058	926	896	896
query37	148	89	85	85
query38	4067	3784	3915	3784
query39	1451	1381	1381	1381
query40	203	118	114	114
query41	50	46	47	46
query42	112	100	96	96
query43	511	467	454	454
query44	1129	764	745	745
query45	195	166	163	163
query46	1093	747	723	723
query47	1913	1770	1806	1770
query48	353	293	286	286
query49	756	430	448	430
query50	827	419	411	411
query51	7014	6931	6935	6931
query52	101	87	86	86
query53	248	176	173	173
query54	561	463	450	450
query55	79	74	74	74
query56	276	257	282	257
query57	1188	1073	1073	1073
query58	232	234	231	231
query59	2967	2893	2692	2692
query60	294	272	273	272
query61	146	103	105	103
query62	729	633	660	633
query63	218	190	183	183
query64	2790	676	663	663
query65	3250	3136	3140	3136
query66	635	335	344	335
query67	15447	15295	15107	15107
query68	2995	562	562	562
query69	405	285	323	285
query70	1181	1129	1129	1129
query71	327	274	268	268
query72	5843	3997	3997	3997
query73	744	320	320	320
query74	9120	8780	8890	8780
query75	3355	2692	2644	2644
query76	1418	969	1031	969
query77	536	321	319	319
query78	9768	9085	9657	9085
query79	1062	534	522	522
query80	692	512	516	512
query81	544	231	233	231
query82	251	139	144	139
query83	173	154	152	152
query84	270	75	77	75
query85	679	288	274	274
query86	302	305	270	270
query87	4428	4243	4349	4243
query88	3135	2314	2314	2314
query89	386	282	280	280
query90	1874	194	191	191
query91	128	104	98	98
query92	58	49	52	49
query93	1051	524	530	524
query94	554	301	307	301
query95	366	271	329	271
query96	585	265	264	264
query97	3199	3075	3095	3075
query98	213	203	194	194
query99	1627	1264	1269	1264
Total cold run time: 305582 ms
Total hot run time: 192514 ms

@doris-robot
Copy link

doris-robot commented Sep 8, 2024

ClickBench: Total hot run time: 31.54 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 64d7fc4ed979eda13ea87e609f407a375dc89ad4, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.04	0.05
query4	1.68	0.08	0.08
query5	0.51	0.50	0.49
query6	1.13	0.73	0.73
query7	0.02	0.02	0.02
query8	0.06	0.05	0.04
query9	0.52	0.50	0.49
query10	0.53	0.55	0.54
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.59
query14	1.40	1.43	1.42
query15	0.86	0.82	0.82
query16	0.35	0.35	0.37
query17	0.94	0.97	0.99
query18	0.22	0.21	0.19
query19	1.79	1.76	1.75
query20	0.01	0.02	0.01
query21	15.39	0.68	0.67
query22	3.80	7.61	2.18
query23	18.29	1.48	1.28
query24	2.12	0.22	0.22
query25	0.16	0.08	0.07
query26	0.26	0.18	0.18
query27	0.07	0.08	0.07
query28	13.29	1.00	0.99
query29	12.65	3.29	3.29
query30	0.24	0.06	0.06
query31	2.90	0.39	0.39
query32	3.23	0.49	0.47
query33	3.03	2.99	2.99
query34	17.16	4.35	4.38
query35	4.44	4.41	4.41
query36	0.65	0.48	0.49
query37	0.19	0.15	0.16
query38	0.16	0.14	0.15
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.71 s
Total hot run time: 31.54 s

dataroaring
dataroaring approved these changes Sep 10, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 10, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Sep 10, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 10, 2024

PR approved by anyone and no changes requested.

@gavinchou gavinchou merged commit 4949a9e into apache:master Sep 10, 2024
gavinchou added a commit that referenced this pull request Sep 11, 2024
@gavinchou
[opt](vault) Do not use latest_fs() in vault mode (#40516)
645755c 
To prevent incorrect storage backends selected by loading data, e.g.
empty vault id passed from FE, we should not use latest_fs() in vault
mode.
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

+1 more reviewer

@Yukang-Lian Yukang-Lian Yukang-Lian approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@gavinchou @doris-robot @Yukang-Lian @dataroaring