Skip to content

[bugfix](hive)Prevent multiple fs from being generated #36954

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Jul 2, 2024
Merged

[bugfix](hive)Prevent multiple fs from being generated #36954

morningman merged 1 commit into from
Jul 2, 2024

Conversation

@wuwenchi
Copy link
Contributor

@wuwenchi wuwenchi commented Jun 27, 2024

Proposed changes

If the user has already registered, there is no need to register again, otherwise fs will generate multiple instances.

@doris-robot
Copy link

doris-robot commented Jun 27, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wuwenchi
Copy link
Contributor Author

wuwenchi commented Jun 27, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jun 27, 2024

TPC-H: Total hot run time: 40121 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fe41acb5c1a486a7cc9377af5b1d32f9fcf32a51, data reload: false

------ Round 1 ----------------------------------
q1	17611	4429	4278	4278
q2	2012	193	191	191
q3	10478	1253	1158	1158
q4	10186	824	753	753
q5	7490	2673	2636	2636
q6	217	132	134	132
q7	949	596	595	595
q8	9240	2096	2094	2094
q9	8936	6525	6511	6511
q10	8902	3719	3724	3719
q11	454	242	234	234
q12	408	235	220	220
q13	17793	2990	2953	2953
q14	262	220	220	220
q15	517	469	489	469
q16	494	368	371	368
q17	978	715	741	715
q18	8081	7545	7502	7502
q19	5896	1491	1493	1491
q20	656	325	321	321
q21	4978	3223	3969	3223
q22	397	338	342	338
Total cold run time: 116935 ms
Total hot run time: 40121 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4402	4262	4310	4262
q2	376	265	258	258
q3	3036	2936	2950	2936
q4	1998	1679	1720	1679
q5	5590	5522	5493	5493
q6	226	125	125	125
q7	2284	1911	1869	1869
q8	3291	3452	3426	3426
q9	8744	8787	8826	8787
q10	4191	3696	3836	3696
q11	595	494	489	489
q12	795	677	598	598
q13	17309	3164	3147	3147
q14	300	271	263	263
q15	535	492	484	484
q16	501	424	435	424
q17	1809	1511	1496	1496
q18	8253	7935	8019	7935
q19	1802	1784	1520	1520
q20	2124	1879	1883	1879
q21	5114	4838	4920	4838
q22	679	549	561	549
Total cold run time: 73954 ms
Total hot run time: 56153 ms

@doris-robot
Copy link

doris-robot commented Jun 27, 2024

TPC-DS: Total hot run time: 175045 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fe41acb5c1a486a7cc9377af5b1d32f9fcf32a51, data reload: false

query1	954	402	398	398
query2	6459	2546	2480	2480
query3	6640	210	220	210
query4	19626	17969	17411	17411
query5	3630	483	469	469
query6	268	186	157	157
query7	4580	292	287	287
query8	311	290	289	289
query9	8558	2447	2423	2423
query10	569	296	285	285
query11	10547	10076	10105	10076
query12	115	87	84	84
query13	1643	365	375	365
query14	9343	7244	7159	7159
query15	237	191	190	190
query16	7678	285	264	264
query17	1901	565	516	516
query18	1854	285	278	278
query19	196	150	147	147
query20	88	81	81	81
query21	218	134	133	133
query22	4382	3991	4005	3991
query23	33906	33554	33640	33554
query24	11803	2953	2877	2877
query25	669	370	368	368
query26	1698	155	158	155
query27	3039	331	329	329
query28	7501	2165	2135	2135
query29	1060	670	660	660
query30	265	165	162	162
query31	991	776	768	768
query32	98	54	53	53
query33	762	287	315	287
query34	1066	473	489	473
query35	765	677	660	660
query36	1118	1004	1002	1002
query37	284	78	73	73
query38	2982	2864	2899	2864
query39	887	830	849	830
query40	260	127	130	127
query41	58	51	55	51
query42	106	106	103	103
query43	600	561	544	544
query44	1207	739	722	722
query45	191	167	163	163
query46	1085	735	719	719
query47	1869	1771	1793	1771
query48	370	303	307	303
query49	954	399	416	399
query50	768	390	387	387
query51	6975	6684	6822	6684
query52	103	95	89	89
query53	356	284	288	284
query54	927	438	439	438
query55	73	72	72	72
query56	276	267	285	267
query57	1155	1091	1070	1070
query58	251	241	236	236
query59	3512	3364	3138	3138
query60	289	268	269	268
query61	99	88	93	88
query62	617	468	468	468
query63	322	293	290	290
query64	9826	2340	1743	1743
query65	3211	3151	3143	3143
query66	1225	328	328	328
query67	15574	14938	14842	14842
query68	7385	535	550	535
query69	628	438	430	430
query70	1257	1160	1163	1160
query71	508	278	278	278
query72	7117	5610	5855	5610
query73	804	325	325	325
query74	6012	5507	5499	5499
query75	4206	2668	2669	2668
query76	4490	940	947	940
query77	667	299	304	299
query78	10968	9921	9931	9921
query79	2246	505	509	505
query80	1246	463	465	463
query81	570	237	228	228
query82	707	102	104	102
query83	195	170	165	165
query84	278	84	93	84
query85	1379	320	270	270
query86	417	314	317	314
query87	3328	3106	3123	3106
query88	3556	2435	2439	2435
query89	475	371	371	371
query90	1824	195	184	184
query91	128	99	102	99
query92	56	48	48	48
query93	2470	508	499	499
query94	1012	191	183	183
query95	411	313	323	313
query96	582	280	269	269
query97	3235	3029	3068	3029
query98	227	210	202	202
query99	1208	860	831	831
Total cold run time: 280542 ms
Total hot run time: 175045 ms

@doris-robot
Copy link

doris-robot commented Jun 27, 2024

ClickBench: Total hot run time: 30.56 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fe41acb5c1a486a7cc9377af5b1d32f9fcf32a51, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.06	0.06
query4	1.66	0.07	0.08
query5	0.50	0.50	0.48
query6	1.13	0.72	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.55	0.50	0.50
query10	0.55	0.54	0.53
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.58	0.58	0.60
query14	0.77	0.76	0.78
query15	0.84	0.81	0.80
query16	0.36	0.37	0.37
query17	1.00	1.03	1.04
query18	0.21	0.27	0.27
query19	1.93	1.73	1.67
query20	0.01	0.01	0.01
query21	15.43	0.77	0.67
query22	4.63	6.79	1.94
query23	18.28	1.34	1.23
query24	2.14	0.23	0.21
query25	0.16	0.09	0.09
query26	0.27	0.17	0.18
query27	0.08	0.08	0.08
query28	13.23	1.02	0.99
query29	12.58	3.25	3.27
query30	0.25	0.06	0.05
query31	2.90	0.38	0.39
query32	3.23	0.46	0.47
query33	2.83	2.90	2.90
query34	17.06	4.43	4.44
query35	4.50	4.46	4.49
query36	0.64	0.46	0.46
query37	0.17	0.15	0.15
query38	0.15	0.15	0.14
query39	0.05	0.04	0.03
query40	0.19	0.14	0.14
query41	0.10	0.04	0.05
query42	0.05	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.78 s
Total hot run time: 30.56 s

morningman
morningman approved these changes Jun 28, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Jun 28, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 28, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jun 28, 2024

PR approved by anyone and no changes requested.

@wuwenchi wuwenchi changed the title [bugfix]Prevent multiple fs from being generated [bugfix](hive)Prevent multiple fs from being generated Jul 2, 2024
wsjz
wsjz approved these changes Jul 2, 2024
View reviewed changes
Copy link
Contributor

@wsjz wsjz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit d33e1a0 into apache:master Jul 2, 2024
@wuwenchi wuwenchi mentioned this pull request Jul 2, 2024
Merged
morningman pushed a commit that referenced this pull request Jul 2, 2024
@wuwenchi
[bugfix](hive)Prevent multiple fs from being generated for 2.1 (#37142)
b3eaf0e 
pick #36954
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@wuwenchi @dataroaring
[bugfix](hive)Prevent multiple fs from being generated (#36954)
3a87ff6 
If the user has already registered, there is no need to register again,
otherwise `fs` will generate multiple instances.
@wuwenchi wuwenchi mentioned this pull request Jul 18, 2024
Merged
morningman pushed a commit that referenced this pull request Jul 23, 2024
@wuwenchi
[bugfix](hive)Prevent multiple fs from being generated for 2.0 (#36954)…
a595705 
… (#38045)

## Proposed changes

bp: #36954
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@wuwenchi
[bugfix](hive)Prevent multiple fs from being generated for 2.0 (apach…
8585e9c 
…e#36954) (apache#38045)

## Proposed changes

bp: apache#36954
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

+1 more reviewer

@wsjz wsjz wsjz approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.14-merged dev/2.1.5-merged dev/3.0.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@wuwenchi @doris-robot @morningman @wsjz @wm1581066 @dataroaring