Skip to content

Conversation

@CalvinKirs
Copy link
Member

PR Description:

Change Background:
The closed field in the current RemoteFileSystem class is static, meaning it is shared globally across all instances of the class. This design leads to issues when multiple parts of the application use RemoteFileSystem and attempt to close it. Once one instance is closed, the static closed field is set to true, causing other instances to incorrectly report that the file system is unavailable. This can cause confusion and inconsistent behavior, especially when multiple threads are involved.

Change Details:

closed Field: The closed field has been moved from a static field to an instance-level field. This ensures that each RemoteFileSystem instance maintains its own shutdown status, eliminating the problem of shared state between instances.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

… Shutdown Issues in RemoteFileSystem

### PR Description:
Change Background:
The closed field in the current RemoteFileSystem class is static, meaning it is shared globally across all instances of the class. This design leads to issues when multiple parts of the application use RemoteFileSystem and attempt to close it. Once one instance is closed, the static closed field is set to true, causing other instances to incorrectly report that the file system is unavailable. This can cause confusion and inconsistent behavior, especially when multiple threads are involved.

### Change Details:
closed Field: The closed field has been moved from a static field to an instance-level field. This ensures that each RemoteFileSystem instance maintains its own shutdown status, eliminating the problem of shared state between instances.
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@CalvinKirs
Copy link
Member Author

run buildall

Copy link
Contributor

@swjtu-zhanglei swjtu-zhanglei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 26, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 31820 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d3cbcb66d15a7d8132367670f6461a27a8941703, data reload: false

------ Round 1 ----------------------------------
q1	17604	5184	5096	5096
q2	2042	291	171	171
q3	10443	1347	732	732
q4	10290	1018	528	528
q5	8457	2442	2442	2442
q6	191	173	137	137
q7	912	748	617	617
q8	9301	1326	1138	1138
q9	5073	4770	4728	4728
q10	6882	2335	1860	1860
q11	489	278	274	274
q12	350	356	215	215
q13	17799	3708	3121	3121
q14	240	229	216	216
q15	528	474	450	450
q16	637	616	594	594
q17	581	887	336	336
q18	6849	6405	6223	6223
q19	1558	971	553	553
q20	316	321	189	189
q21	2793	2310	1902	1902
q22	361	336	298	298
Total cold run time: 103696 ms
Total hot run time: 31820 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5238	5148	5197	5148
q2	237	333	230	230
q3	2176	2710	2285	2285
q4	1459	1837	1404	1404
q5	4314	4162	4195	4162
q6	207	167	127	127
q7	1883	1906	1808	1808
q8	2666	2644	2666	2644
q9	7251	7156	7140	7140
q10	3026	3220	2754	2754
q11	573	499	487	487
q12	671	767	653	653
q13	3621	3912	3299	3299
q14	285	295	269	269
q15	521	472	460	460
q16	651	688	663	663
q17	1150	1658	1316	1316
q18	7724	7376	7151	7151
q19	827	839	974	839
q20	1983	2046	1901	1901
q21	5480	4996	5030	4996
q22	634	624	575	575
Total cold run time: 52577 ms
Total hot run time: 50311 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184182 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d3cbcb66d15a7d8132367670f6461a27a8941703, data reload: false

query1	979	396	384	384
query2	6518	1856	1876	1856
query3	6792	219	220	219
query4	26535	23714	22983	22983
query5	4359	666	550	550
query6	299	208	193	193
query7	4610	510	292	292
query8	305	241	230	230
query9	8599	2600	2600	2600
query10	471	319	255	255
query11	15660	15213	15046	15046
query12	163	120	108	108
query13	1659	525	408	408
query14	9456	6599	6599	6599
query15	217	204	176	176
query16	7468	649	463	463
query17	1219	726	561	561
query18	1977	399	307	307
query19	197	188	156	156
query20	120	114	119	114
query21	206	126	105	105
query22	4112	4186	3939	3939
query23	33784	32997	32835	32835
query24	7745	2399	2404	2399
query25	537	448	384	384
query26	1232	270	156	156
query27	2155	508	323	323
query28	3915	2430	2431	2430
query29	733	552	424	424
query30	235	186	158	158
query31	936	835	751	751
query32	76	68	65	65
query33	559	361	312	312
query34	798	855	492	492
query35	783	801	723	723
query36	938	968	880	880
query37	114	96	76	76
query38	4124	4254	4157	4157
query39	1470	1395	1441	1395
query40	203	112	104	104
query41	61	55	66	55
query42	122	106	101	101
query43	490	515	507	507
query44	1279	798	793	793
query45	177	163	159	159
query46	856	1041	640	640
query47	1738	1785	1743	1743
query48	370	403	292	292
query49	803	514	432	432
query50	686	731	412	412
query51	4199	4317	4173	4173
query52	109	115	95	95
query53	251	253	187	187
query54	490	485	401	401
query55	80	78	78	78
query56	270	295	258	258
query57	1127	1156	1054	1054
query58	248	239	243	239
query59	2676	2783	2698	2698
query60	278	270	263	263
query61	117	114	118	114
query62	813	742	677	677
query63	233	194	185	185
query64	4360	998	707	707
query65	3230	3145	3149	3145
query66	1120	406	312	312
query67	15637	15626	15114	15114
query68	8230	932	516	516
query69	471	305	265	265
query70	1183	1114	1100	1100
query71	456	287	252	252
query72	5575	3529	3716	3529
query73	789	707	348	348
query74	9015	9244	9098	9098
query75	3775	3206	2748	2748
query76	3748	1168	741	741
query77	776	382	280	280
query78	10051	10029	9287	9287
query79	2766	850	628	628
query80	656	555	459	459
query81	524	283	242	242
query82	724	124	93	93
query83	179	171	166	166
query84	254	90	132	90
query85	809	359	300	300
query86	382	304	288	288
query87	4496	4652	4332	4332
query88	3697	2213	2200	2200
query89	394	319	286	286
query90	1839	197	199	197
query91	139	137	116	116
query92	75	62	61	61
query93	1925	1048	589	589
query94	666	413	300	300
query95	360	269	260	260
query96	485	560	268	268
query97	3310	3403	3257	3257
query98	230	209	207	207
query99	1323	1403	1237	1237
Total cold run time: 273380 ms
Total hot run time: 184182 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.69 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d3cbcb66d15a7d8132367670f6461a27a8941703, data reload: false

query1	0.03	0.05	0.03
query2	0.07	0.04	0.03
query3	0.24	0.07	0.07
query4	1.61	0.10	0.11
query5	0.56	0.56	0.56
query6	1.18	0.72	0.72
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.57	0.55	0.51
query10	0.58	0.58	0.57
query11	0.15	0.11	0.11
query12	0.15	0.11	0.12
query13	0.63	0.60	0.60
query14	2.68	2.70	2.67
query15	0.93	0.86	0.85
query16	0.37	0.37	0.38
query17	1.03	1.04	1.04
query18	0.22	0.20	0.19
query19	1.89	1.74	2.01
query20	0.02	0.01	0.01
query21	15.35	0.89	0.56
query22	0.76	1.30	0.70
query23	14.79	1.39	0.58
query24	7.38	1.48	0.82
query25	0.50	0.20	0.12
query26	0.69	0.17	0.16
query27	0.05	0.05	0.05
query28	9.50	0.85	0.41
query29	12.57	3.96	3.28
query30	0.25	0.09	0.06
query31	2.83	0.59	0.39
query32	3.23	0.56	0.48
query33	3.06	3.00	3.02
query34	15.87	5.12	4.47
query35	4.57	4.51	4.54
query36	0.67	0.49	0.48
query37	0.09	0.06	0.07
query38	0.05	0.03	0.04
query39	0.03	0.02	0.03
query40	0.16	0.14	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.02
Total cold run time: 105.53 s
Total hot run time: 30.69 s

@CalvinKirs
Copy link
Member Author

run p0

@CalvinKirs
Copy link
Member Author

run cloudp0

@CalvinKirs
Copy link
Member Author

run p0

@CalvinKirs CalvinKirs merged commit 20d95aa into apache:master Feb 26, 2025
32 checks passed
@CalvinKirs CalvinKirs deleted the master-kerberos-test-case branch February 26, 2025 10:01
github-actions bot pushed a commit that referenced this pull request Feb 26, 2025
… Shutdown Issues in RemoteFileSystem (#48351)

### PR Description:
Change Background:
The closed field in the current RemoteFileSystem class is static,
meaning it is shared globally across all instances of the class. This
design leads to issues when multiple parts of the application use
RemoteFileSystem and attempt to close it. Once one instance is closed,
the static closed field is set to true, causing other instances to
incorrectly report that the file system is unavailable. This can cause
confusion and inconsistent behavior, especially when multiple threads
are involved.

### Change Details:
closed Field: The closed field has been moved from a static field to an
instance-level field. This ensures that each RemoteFileSystem instance
maintains its own shutdown status, eliminating the problem of shared
state between instances.
github-actions bot pushed a commit that referenced this pull request Feb 26, 2025
… Shutdown Issues in RemoteFileSystem (#48351)

### PR Description:
Change Background:
The closed field in the current RemoteFileSystem class is static,
meaning it is shared globally across all instances of the class. This
design leads to issues when multiple parts of the application use
RemoteFileSystem and attempt to close it. Once one instance is closed,
the static closed field is set to true, causing other instances to
incorrectly report that the file system is unavailable. This can cause
confusion and inconsistent behavior, especially when multiple threads
are involved.

### Change Details:
closed Field: The closed field has been moved from a static field to an
instance-level field. This ensures that each RemoteFileSystem instance
maintains its own shutdown status, eliminating the problem of shared
state between instances.
morningman pushed a commit that referenced this pull request Feb 26, 2025
…Avoid Global Shutdown Issues in RemoteFileSystem #48351 (#48379)

Cherry-picked from #48351

Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
morningman pushed a commit that referenced this pull request Feb 27, 2025
…Avoid Global Shutdown Issues in RemoteFileSystem #48351 (#48378)

Cherry-picked from #48351

Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
zhiqiang-hhhh pushed a commit to zhiqiang-hhhh/doris that referenced this pull request Feb 27, 2025
… Shutdown Issues in RemoteFileSystem (apache#48351)

### PR Description:
Change Background:
The closed field in the current RemoteFileSystem class is static,
meaning it is shared globally across all instances of the class. This
design leads to issues when multiple parts of the application use
RemoteFileSystem and attempt to close it. Once one instance is closed,
the static closed field is set to true, causing other instances to
incorrectly report that the file system is unavailable. This can cause
confusion and inconsistent behavior, especially when multiple threads
are involved.

### Change Details:
closed Field: The closed field has been moved from a static field to an
instance-level field. This ensures that each RemoteFileSystem instance
maintains its own shutdown status, eliminating the problem of shared
state between instances.
seawinde pushed a commit to seawinde/doris that referenced this pull request Feb 28, 2025
… Shutdown Issues in RemoteFileSystem (apache#48351)

### PR Description:
Change Background:
The closed field in the current RemoteFileSystem class is static,
meaning it is shared globally across all instances of the class. This
design leads to issues when multiple parts of the application use
RemoteFileSystem and attempt to close it. Once one instance is closed,
the static closed field is set to true, causing other instances to
incorrectly report that the file system is unavailable. This can cause
confusion and inconsistent behavior, especially when multiple threads
are involved.

### Change Details:
closed Field: The closed field has been moved from a static field to an
instance-level field. This ensures that each RemoteFileSystem instance
maintains its own shutdown status, eliminating the problem of shared
state between instances.
mymeiyi pushed a commit to mymeiyi/doris that referenced this pull request Mar 4, 2025
… Shutdown Issues in RemoteFileSystem (apache#48351)

### PR Description:
Change Background:
The closed field in the current RemoteFileSystem class is static,
meaning it is shared globally across all instances of the class. This
design leads to issues when multiple parts of the application use
RemoteFileSystem and attempt to close it. Once one instance is closed,
the static closed field is set to true, causing other instances to
incorrectly report that the file system is unavailable. This can cause
confusion and inconsistent behavior, especially when multiple threads
are involved.

### Change Details:
closed Field: The closed field has been moved from a static field to an
instance-level field. This ensures that each RemoteFileSystem instance
maintains its own shutdown status, eliminating the problem of shared
state between instances.
@gavinchou gavinchou mentioned this pull request Apr 23, 2025
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
… Shutdown Issues in RemoteFileSystem (apache#48351)

### PR Description:
Change Background:
The closed field in the current RemoteFileSystem class is static,
meaning it is shared globally across all instances of the class. This
design leads to issues when multiple parts of the application use
RemoteFileSystem and attempt to close it. Once one instance is closed,
the static closed field is set to true, causing other instances to
incorrectly report that the file system is unavailable. This can cause
confusion and inconsistent behavior, especially when multiple threads
are involved.

### Change Details:
closed Field: The closed field has been moved from a static field to an
instance-level field. This ensures that each RemoteFileSystem instance
maintains its own shutdown status, eliminating the problem of shared
state between instances.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.9-merged dev/3.0.5-merged p0_b reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants