Skip to content

branch-3.0: [Fix](remote-fs)Change closed Field to Instance-Level to Avoid Global Shutdown Issues in RemoteFileSystem #48351 #48378

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Feb 27, 2025
Merged

branch-3.0: [Fix](remote-fs)Change closed Field to Instance-Level to Avoid Global Shutdown Issues in RemoteFileSystem #48351 #48378

morningman merged 1 commit into from
Feb 27, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Feb 26, 2025

Cherry-picked from #48351

@CalvinKirs
[Fix](remote-fs)Change closed Field to Instance-Level to Avoid Global…
37d15b6 
… Shutdown Issues in RemoteFileSystem (#48351)

### PR Description:
Change Background:
The closed field in the current RemoteFileSystem class is static,
meaning it is shared globally across all instances of the class. This
design leads to issues when multiple parts of the application use
RemoteFileSystem and attempt to close it. Once one instance is closed,
the static closed field is set to true, causing other instances to
incorrectly report that the file system is unavailable. This can cause
confusion and inconsistent behavior, especially when multiple threads
are involved.

### Change Details:
closed Field: The closed field has been moved from a static field to an
instance-level field. This ensures that each RemoteFileSystem instance
maintains its own shutdown status, eliminating the problem of shared
state between instances.
@hello-stephen
Copy link
Contributor

hello-stephen commented Feb 26, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Feb 26, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Feb 26, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Feb 26, 2025

TPC-H: Total hot run time: 39983 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 37d15b66cad33bfe28459f8af7086c9daeb68a49, data reload: false

------ Round 1 ----------------------------------
q1	17585	6715	6592	6592
q2	2050	169	182	169
q3	10576	1048	1146	1048
q4	10564	763	667	667
q5	7780	2880	2857	2857
q6	230	135	131	131
q7	972	612	602	602
q8	9367	1957	2007	1957
q9	6561	6360	6398	6360
q10	7040	2248	2265	2248
q11	463	267	259	259
q12	401	213	208	208
q13	17787	2988	3027	2988
q14	242	215	220	215
q15	505	463	471	463
q16	666	575	579	575
q17	963	570	510	510
q18	7372	6742	6689	6689
q19	1442	1019	1121	1019
q20	457	211	196	196
q21	3985	3234	3326	3234
q22	1099	998	996	996
Total cold run time: 108107 ms
Total hot run time: 39983 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6579	6559	6539	6539
q2	329	237	237	237
q3	2913	2751	2910	2751
q4	2051	1799	1806	1799
q5	5716	5754	5791	5754
q6	212	130	129	129
q7	2203	1823	1839	1823
q8	3334	3559	3512	3512
q9	8933	8866	8907	8866
q10	3594	3544	3513	3513
q11	582	491	511	491
q12	798	616	642	616
q13	9829	3259	3168	3168
q14	305	285	267	267
q15	519	467	453	453
q16	686	654	671	654
q17	1840	1646	1622	1622
q18	8208	7741	7712	7712
q19	1672	1527	1560	1527
q20	2084	1867	1865	1865
q21	5558	5330	5344	5330
q22	1157	1111	1005	1005
Total cold run time: 69102 ms
Total hot run time: 59633 ms

@doris-robot
Copy link

doris-robot commented Feb 26, 2025

TPC-DS: Total hot run time: 199846 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 37d15b66cad33bfe28459f8af7086c9daeb68a49, data reload: false

query1	1289	937	908	908
query2	6264	2051	1959	1959
query3	10976	4522	4570	4522
query4	60966	29584	23352	23352
query5	5193	475	471	471
query6	412	182	200	182
query7	5469	320	308	308
query8	313	239	234	234
query9	8402	2671	2638	2638
query10	458	277	291	277
query11	17595	15084	15759	15084
query12	175	109	109	109
query13	1413	457	457	457
query14	10680	8108	8116	8108
query15	216	181	192	181
query16	7010	500	501	500
query17	1165	607	607	607
query18	1940	338	319	319
query19	218	172	164	164
query20	124	119	116	116
query21	209	107	105	105
query22	4789	4480	4569	4480
query23	34960	34247	34428	34247
query24	6233	2988	3096	2988
query25	553	425	443	425
query26	657	177	179	177
query27	1846	365	378	365
query28	4371	2464	2507	2464
query29	731	466	441	441
query30	245	159	166	159
query31	1041	831	843	831
query32	68	60	57	57
query33	485	304	303	303
query34	945	511	538	511
query35	874	743	720	720
query36	1106	951	979	951
query37	119	72	70	70
query38	4172	4044	4064	4044
query39	1509	1504	1473	1473
query40	222	103	105	103
query41	53	54	48	48
query42	118	98	105	98
query43	546	497	493	493
query44	1220	846	853	846
query45	188	167	169	167
query46	1178	783	755	755
query47	2025	1876	1932	1876
query48	486	393	403	393
query49	767	408	422	408
query50	874	456	419	419
query51	7433	7266	7302	7266
query52	104	92	90	90
query53	274	191	189	189
query54	566	450	487	450
query55	84	83	83	83
query56	276	286	270	270
query57	1280	1225	1216	1216
query58	229	216	225	216
query59	3354	3195	2961	2961
query60	292	262	261	261
query61	113	109	110	109
query62	761	691	685	685
query63	223	192	195	192
query64	1401	698	653	653
query65	3330	3228	3231	3228
query66	718	301	313	301
query67	16125	15620	15847	15620
query68	3957	618	585	585
query69	422	271	273	271
query70	1223	1034	1126	1034
query71	361	264	262	262
query72	6361	4088	4094	4088
query73	789	349	367	349
query74	10481	9261	9347	9261
query75	3385	2659	2660	2659
query76	2084	1047	1099	1047
query77	556	280	278	278
query78	10608	9620	9733	9620
query79	2047	609	614	609
query80	1332	452	439	439
query81	527	247	242	242
query82	1232	92	91	91
query83	176	153	145	145
query84	287	85	77	77
query85	1007	313	304	304
query86	389	296	302	296
query87	4549	4309	4301	4301
query88	3723	2396	2362	2362
query89	434	294	294	294
query90	1980	195	201	195
query91	204	151	152	151
query92	62	56	55	55
query93	2849	562	553	553
query94	772	302	303	302
query95	376	282	304	282
query96	648	288	278	278
query97	3353	3231	3215	3215
query98	219	209	203	203
query99	1608	1295	1309	1295
Total cold run time: 317432 ms
Total hot run time: 199846 ms

@doris-robot
Copy link

doris-robot commented Feb 26, 2025

ClickBench: Total hot run time: 33.75 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 37d15b66cad33bfe28459f8af7086c9daeb68a49, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.11	0.11
query5	0.54	0.50	0.53
query6	1.14	0.73	0.73
query7	0.03	0.02	0.02
query8	0.04	0.04	0.04
query9	0.59	0.50	0.49
query10	0.56	0.55	0.56
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.60	0.60
query14	2.72	2.85	2.83
query15	0.91	0.83	0.84
query16	0.43	0.38	0.39
query17	1.01	1.00	1.05
query18	0.23	0.23	0.22
query19	1.91	1.80	2.02
query20	0.01	0.01	0.02
query21	15.36	0.58	0.58
query22	2.66	2.61	2.37
query23	16.97	0.91	0.84
query24	3.29	1.89	1.77
query25	0.27	0.11	0.05
query26	0.51	0.13	0.14
query27	0.04	0.04	0.04
query28	8.63	0.55	0.55
query29	12.56	3.30	3.26
query30	0.25	0.06	0.08
query31	2.85	0.38	0.38
query32	3.24	0.46	0.46
query33	3.03	2.98	3.04
query34	17.08	4.48	4.56
query35	4.64	4.59	4.58
query36	0.67	0.46	0.49
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.16	0.12	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.51 s
Total hot run time: 33.75 s

dataroaring
dataroaring approved these changes Feb 26, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 26, 2025
@github-actions
Copy link
Contributor Author

github-actions bot commented Feb 26, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor Author

github-actions bot commented Feb 26, 2025

PR approved by anyone and no changes requested.

@morningman morningman merged commit 1799787 into branch-3.0 Feb 27, 2025
23 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-48351-branch-3.0 branch February 27, 2025 02:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@hello-stephen @doris-robot @dataroaring @morningman @CalvinKirs