Skip to content

branch-3.0: [fix](create-resource) fix potential concurrent modification exception when creating resource (#50356, #50615) #50533

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 2 commits into from
May 15, 2025
Merged

branch-3.0: [fix](create-resource) fix potential concurrent modification exception when creating resource (#50356, #50615) #50533

dataroaring merged 2 commits into from
May 15, 2025

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Apr 29, 2025 •
edited
Loading

@morningman morningman requested a review from dataroaring as a code owner April 29, 2025 06:25
@hello-stephen
Copy link
Contributor

hello-stephen commented Apr 29, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman
Copy link
Contributor Author

morningman commented Apr 29, 2025

run buildall

@morningman morningman changed the title [fix](create-resource) fix potential concurrent modification exception when creating resource (#50356) [fix](create-resource) fix potential concurrent modification exception when creating resource (#50356, #50615) May 7, 2025
@morningman
Copy link
Contributor Author

morningman commented May 7, 2025

run buildall

gavinchou
gavinchou previously approved these changes May 7, 2025
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 7, 2025
@github-actions
Copy link
Contributor

github-actions bot commented May 7, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented May 7, 2025

PR approved by anyone and no changes requested.

@morningman
Copy link
Contributor Author

morningman commented May 7, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 7, 2025

TPC-H: Total hot run time: 40004 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6c52b12e931315e484b90af0e08df1791c0d729b, data reload: false

------ Round 1 ----------------------------------
q1	17599	6791	6622	6622
q2	2069	180	170	170
q3	10614	1047	1191	1047
q4	10564	692	719	692
q5	7735	2815	2828	2815
q6	223	136	131	131
q7	975	610	605	605
q8	9347	1914	1954	1914
q9	6582	6366	6413	6366
q10	7026	2237	2286	2237
q11	459	246	259	246
q12	398	213	207	207
q13	17802	2945	2971	2945
q14	235	203	221	203
q15	483	455	468	455
q16	688	584	584	584
q17	973	606	602	602
q18	7235	6817	6694	6694
q19	1401	1053	1031	1031
q20	470	206	212	206
q21	4168	3241	3321	3241
q22	1077	991	1001	991
Total cold run time: 108123 ms
Total hot run time: 40004 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6567	6599	6553	6553
q2	323	228	231	228
q3	2892	2747	2864	2747
q4	2051	1836	1799	1799
q5	5765	5713	5762	5713
q6	212	130	134	130
q7	2242	1896	1863	1863
q8	3385	3614	3540	3540
q9	9025	8990	8958	8958
q10	3601	3547	3540	3540
q11	603	504	500	500
q12	803	636	634	634
q13	8053	3196	3166	3166
q14	296	261	266	261
q15	507	469	465	465
q16	667	666	660	660
q17	1835	1732	1594	1594
q18	8183	7698	7726	7698
q19	1661	1478	1520	1478
q20	2062	1906	1858	1858
q21	5489	5238	5192	5192
q22	1140	1048	1031	1031
Total cold run time: 67362 ms
Total hot run time: 59608 ms

@doris-robot
Copy link

doris-robot commented May 7, 2025

TPC-DS: Total hot run time: 196614 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6c52b12e931315e484b90af0e08df1791c0d729b, data reload: false

query1	1328	915	880	880
query2	6309	2011	1977	1977
query3	10824	4103	4212	4103
query4	61322	29912	23670	23670
query5	5273	481	450	450
query6	394	180	181	180
query7	5477	307	302	302
query8	307	225	228	225
query9	8438	2592	2555	2555
query10	466	254	257	254
query11	17480	15204	15728	15204
query12	160	105	102	102
query13	1425	435	424	424
query14	10460	7167	6619	6619
query15	214	189	177	177
query16	6970	479	486	479
query17	1157	596	578	578
query18	1651	327	316	316
query19	210	159	160	159
query20	121	117	108	108
query21	206	102	106	102
query22	4717	4546	4733	4546
query23	35285	33949	33791	33791
query24	6180	2922	2937	2922
query25	550	417	419	417
query26	651	173	168	168
query27	1852	379	349	349
query28	4450	2466	2475	2466
query29	712	472	438	438
query30	236	163	165	163
query31	1033	818	796	796
query32	73	54	56	54
query33	411	270	280	270
query34	916	508	509	508
query35	868	765	731	731
query36	1069	944	957	944
query37	123	64	70	64
query38	4038	3974	4001	3974
query39	1539	1505	1460	1460
query40	215	97	95	95
query41	52	50	51	50
query42	114	100	98	98
query43	515	483	491	483
query44	1140	805	822	805
query45	188	178	179	178
query46	1147	711	728	711
query47	2042	1956	1965	1956
query48	470	383	385	383
query49	754	403	401	401
query50	821	435	430	430
query51	7470	7276	7299	7276
query52	102	92	93	92
query53	256	179	187	179
query54	569	454	449	449
query55	84	82	84	82
query56	273	263	259	259
query57	1277	1186	1158	1158
query58	224	209	228	209
query59	3151	2976	3182	2976
query60	287	272	284	272
query61	119	113	123	113
query62	786	677	680	677
query63	215	180	183	180
query64	1323	673	651	651
query65	3277	3190	3190	3190
query66	650	294	308	294
query67	16009	15471	15702	15471
query68	4141	569	573	569
query69	424	262	258	258
query70	1210	1131	1070	1070
query71	325	254	243	243
query72	6279	3939	4061	3939
query73	748	348	371	348
query74	10345	9273	9330	9273
query75	3364	2639	2661	2639
query76	1968	1080	1055	1055
query77	498	274	267	267
query78	10770	9789	9552	9552
query79	1803	598	602	598
query80	1085	429	427	427
query81	518	245	237	237
query82	1253	87	85	85
query83	167	136	146	136
query84	280	77	77	77
query85	948	302	285	285
query86	347	284	295	284
query87	4419	4279	4230	4230
query88	3878	2355	2350	2350
query89	423	292	287	287
query90	1899	184	182	182
query91	184	143	146	143
query92	67	48	51	48
query93	1954	539	545	539
query94	738	287	274	274
query95	348	252	255	252
query96	620	285	278	278
query97	3296	3168	3163	3163
query98	210	203	200	200
query99	1575	1283	1310	1283
Total cold run time: 314509 ms
Total hot run time: 196614 ms

@doris-robot
Copy link

doris-robot commented May 7, 2025

ClickBench: Total hot run time: 33.12 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6c52b12e931315e484b90af0e08df1791c0d729b, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.11
query5	0.53	0.51	0.51
query6	1.14	0.73	0.74
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.55	0.51	0.50
query10	0.57	0.55	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.59	0.59
query14	2.71	2.72	2.73
query15	0.90	0.82	0.82
query16	0.38	0.38	0.37
query17	1.06	1.04	1.06
query18	0.23	0.22	0.22
query19	1.92	1.80	2.06
query20	0.01	0.01	0.01
query21	15.44	0.59	0.59
query22	2.90	2.85	2.15
query23	16.94	0.99	0.82
query24	3.08	1.71	2.13
query25	0.19	0.04	0.16
query26	0.70	0.13	0.14
query27	0.04	0.04	0.03
query28	8.66	0.54	0.49
query29	12.55	3.26	3.26
query30	0.24	0.06	0.06
query31	2.85	0.38	0.38
query32	3.26	0.47	0.45
query33	2.95	3.02	2.99
query34	17.26	4.45	4.42
query35	4.54	4.49	4.50
query36	0.66	0.48	0.50
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.67 s
Total hot run time: 33.12 s

@morningman
Copy link
Contributor Author

morningman commented May 7, 2025

run buildall

@morningman morningman changed the title [fix](create-resource) fix potential concurrent modification exception when creating resource (#50356, #50615) branch-3.0: [fix](create-resource) fix potential concurrent modification exception when creating resource (#50356, #50615) May 7, 2025
morningman and others added 2 commits May 14, 2025 10:30
@morningman
[fix](create-resource) fix potential concurrent modification exceptio…
1c5ff56 
…n when creating resource (apache#50356)

Related PR: apache#48424

Problem Summary:

The `CreateResourceCommandTest` may throw concurrent modification
exception.
This is because in `JdbcResource`, the origin property map may be
modified.

This PR changes；
1. Change the origin property map to ImmutableMap, to avoid potential
exception.
2. Copy the map to the jdbc resource to avoid modifying the origin map
@yagagagaga @morningman
[fix](vault) fix UnsupportedOperationException if no use_path_style…
d3608b4 
… in CreateStorageVaultStmt (apache#50615)

Related PR: apache#50356 apache#45155

Problem Summary:

```sql
CREATE STORAGE VAULT test PROPERTIES
    (
      "type"="S3",
      "s3.endpoint"="cos.ap-guangzhou.myqcloud.com",
      "s3.access_key" = "*****",
      "s3.secret_key" = "*****",
      "s3.region" = "ap-guangzhou",
      "s3.root.path" = "prefix",
      "s3.bucket" = "test",
      "provider" = "COS"
    )
```

throw Exception:
```java
2025-05-06 15:05:16,395 WARN (mysql-nio-pool-0|235) [StmtExecutor.handleDdlStmt():3216] DDL statement(CREATE STORAGE VAULT test PROPERTIES
    (
      "type"="S3",
      "s3.endpoint"="cos.ap-guangzhou.myqcloud.com",
      "s3.access_key" = "*****",
      "s3.secret_key" = "*****",
      "s3.region" = "ap-guangzhou",
      "s3.root.path" = "prefix",
      "s3.bucket" = "test",
      "provider" = "COS"
    )) process failed.
java.lang.UnsupportedOperationException: null
        at com.google.common.collect.ImmutableMap.put(ImmutableMap.java:814) ~[guava-33.2.1-jre.jar:?]
        at org.apache.doris.catalog.StorageVault.getStorageVaultInstance(StorageVault.java:151) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.StorageVault.fromStmt(StorageVault.java:103) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.StorageVaultMgr.createStorageVaultResource(StorageVaultMgr.java:69) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.DdlExecutor.execute(DdlExecutor.java:440) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.handleDdlStmt(StmtExecutor.java:3195) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:1124) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:644) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:574) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:559) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:349) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:249) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:233) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:261) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:444) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
```
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label May 14, 2025
@morningman
Copy link
Contributor Author

morningman commented May 14, 2025

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented May 15, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 15, 2025
dataroaring
dataroaring approved these changes May 15, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 3ae4211 into apache:branch-3.0 May 15, 2025
24 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@morningman @hello-stephen @doris-robot @gavinchou @dataroaring @yagagagaga