Skip to content

[fix](iceberg)Fix the failure of creating a table in COS #49885

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Apr 9, 2025
Merged

[fix](iceberg)Fix the failure of creating a table in COS #49885

morningman merged 1 commit into from
Apr 9, 2025

Conversation

@wuwenchi
Copy link
Contributor

@wuwenchi wuwenchi commented Apr 8, 2025

What problem does this PR solve?

Problem Summary:

When use JDK17 with cos, if create an iceberg table, an error will occur:

java.lang.IllegalAccessError: class org.apache.hadoop.fs.cosn.buffer.CosNMappedBuffer (in unnamed module @0x4bb33f74) cannot access class sun.nio.ch.FileChannelImpl (in module java.base) because module java.base does not export
sun.nio.ch to unnamed module @0x4bb33f74

so, we should add JVM config: --add-opens java.base/sun.nio.ch=ALL-UNNAMED.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Apr 8, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@wuwenchi
Copy link
Contributor Author

wuwenchi commented Apr 8, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 8, 2025

TPC-H: Total hot run time: 34282 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6a47d1728dc45d5f7a04d572fb6b4a6a27d9e0ad, data reload: false

------ Round 1 ----------------------------------
q1	25632	5110	5026	5026
q2	2065	316	199	199
q3	10337	1219	676	676
q4	10240	1003	517	517
q5	7537	2305	2312	2305
q6	184	159	130	130
q7	923	745	619	619
q8	9330	1304	1181	1181
q9	7055	5181	5096	5096
q10	6861	2285	1876	1876
q11	481	285	262	262
q12	353	351	220	220
q13	17770	3683	3124	3124
q14	220	218	206	206
q15	534	483	496	483
q16	622	612	604	604
q17	621	840	378	378
q18	7488	7109	7228	7109
q19	1454	956	562	562
q20	329	342	230	230
q21	4411	3679	2496	2496
q22	1021	987	983	983
Total cold run time: 115468 ms
Total hot run time: 34282 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5145	5121	5097	5097
q2	245	335	237	237
q3	2164	2641	2319	2319
q4	1403	1777	1448	1448
q5	4500	4394	4431	4394
q6	215	172	126	126
q7	1994	1931	1774	1774
q8	2630	2477	2473	2473
q9	7256	7169	7094	7094
q10	2955	3168	2758	2758
q11	572	517	501	501
q12	713	781	623	623
q13	3516	3866	3357	3357
q14	295	300	283	283
q15	522	467	476	467
q16	672	696	666	666
q17	1141	1553	1384	1384
q18	7726	7505	7471	7471
q19	807	864	910	864
q20	1917	1986	1844	1844
q21	5329	4873	4856	4856
q22	1074	1072	1021	1021
Total cold run time: 52791 ms
Total hot run time: 51057 ms

@doris-robot
Copy link

doris-robot commented Apr 8, 2025

TPC-DS: Total hot run time: 192579 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6a47d1728dc45d5f7a04d572fb6b4a6a27d9e0ad, data reload: false

query1	1421	1072	1046	1046
query2	6549	1937	1938	1937
query3	11061	4450	4474	4450
query4	25735	23760	22930	22930
query5	4848	626	468	468
query6	305	203	196	196
query7	3987	485	278	278
query8	287	259	232	232
query9	8500	2574	2548	2548
query10	507	310	269	269
query11	15768	15183	14786	14786
query12	163	113	106	106
query13	1555	486	380	380
query14	9533	6135	6168	6135
query15	234	193	166	166
query16	7684	655	485	485
query17	1550	763	594	594
query18	2075	413	317	317
query19	219	186	167	167
query20	125	125	122	122
query21	206	127	112	112
query22	4550	4533	4400	4400
query23	34590	33330	33318	33318
query24	8246	2479	2405	2405
query25	497	484	405	405
query26	723	271	157	157
query27	2622	484	342	342
query28	4709	2424	2412	2412
query29	628	607	471	471
query30	272	231	199	199
query31	906	893	834	834
query32	80	66	63	63
query33	567	403	332	332
query34	799	856	524	524
query35	814	833	784	784
query36	977	1027	932	932
query37	133	114	80	80
query38	4330	4145	4096	4096
query39	1549	1447	1446	1446
query40	224	126	114	114
query41	59	61	61	61
query42	128	110	113	110
query43	518	522	474	474
query44	1294	816	814	814
query45	190	184	175	175
query46	845	1023	630	630
query47	1815	1924	1811	1811
query48	425	404	309	309
query49	709	553	411	411
query50	653	698	407	407
query51	4366	4282	4206	4206
query52	110	111	104	104
query53	226	261	191	191
query54	604	586	523	523
query55	89	82	83	82
query56	310	293	301	293
query57	1187	1194	1130	1130
query58	267	272	265	265
query59	2747	2820	3037	2820
query60	339	347	312	312
query61	132	127	135	127
query62	765	764	693	693
query63	230	188	189	188
query64	2945	1064	704	704
query65	4385	4342	4299	4299
query66	1080	408	320	320
query67	16556	15672	15283	15283
query68	10698	760	508	508
query69	558	309	277	277
query70	1371	1161	1120	1120
query71	490	364	286	286
query72	5992	4799	4904	4799
query73	2741	659	352	352
query74	9250	8844	8902	8844
query75	4647	3268	2670	2670
query76	6210	1193	765	765
query77	1005	380	297	297
query78	9966	9989	9377	9377
query79	9461	738	543	543
query80	714	515	444	444
query81	484	252	224	224
query82	254	127	99	99
query83	271	261	247	247
query84	302	114	87	87
query85	820	357	323	323
query86	332	330	298	298
query87	4642	4629	4312	4312
query88	3062	2233	2210	2210
query89	419	313	282	282
query90	2423	215	215	215
query91	137	148	111	111
query92	72	61	57	57
query93	3363	895	564	564
query94	783	397	314	314
query95	387	300	283	283
query96	481	553	277	277
query97	3128	3146	3142	3142
query98	236	233	196	196
query99	1379	1434	1276	1276
Total cold run time: 298580 ms
Total hot run time: 192579 ms

@doris-robot
Copy link

doris-robot commented Apr 8, 2025

ClickBench: Total hot run time: 31.52 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6a47d1728dc45d5f7a04d572fb6b4a6a27d9e0ad, data reload: false

query1	0.03	0.04	0.03
query2	0.12	0.10	0.10
query3	0.25	0.19	0.20
query4	1.59	0.19	0.19
query5	0.60	0.58	0.57
query6	1.18	0.72	0.71
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.53	0.50
query10	0.62	0.57	0.57
query11	0.15	0.10	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.59
query14	2.68	2.81	2.68
query15	0.95	0.87	0.84
query16	0.40	0.38	0.38
query17	1.04	1.02	1.04
query18	0.21	0.20	0.20
query19	2.01	1.94	1.82
query20	0.02	0.01	0.01
query21	15.35	0.92	0.53
query22	0.77	1.17	0.79
query23	14.79	1.41	0.59
query24	7.18	1.25	1.17
query25	0.49	0.18	0.06
query26	0.60	0.17	0.15
query27	0.06	0.06	0.05
query28	9.93	0.86	0.43
query29	12.54	3.96	3.30
query30	0.24	0.08	0.06
query31	2.82	0.60	0.39
query32	3.23	0.58	0.46
query33	3.02	3.14	3.05
query34	15.86	5.11	4.53
query35	4.50	4.50	4.52
query36	0.69	0.50	0.49
query37	0.09	0.07	0.07
query38	0.06	0.04	0.03
query39	0.02	0.02	0.02
query40	0.17	0.15	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 105.8 s
Total hot run time: 31.52 s

@github-actions
Copy link
Contributor

github-actions bot commented Apr 9, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 9, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Apr 9, 2025

PR approved by anyone and no changes requested.

@morningman morningman merged commit c923206 into apache:master Apr 9, 2025
33 of 35 checks passed
github-actions bot pushed a commit that referenced this pull request Apr 9, 2025
@wuwenchi
[fix](iceberg)Fix the failure of creating a table in COS (#49885)
e7ecc8b 
### What problem does this PR solve?

Problem Summary:

When use JDK17 with cos, if create an iceberg table, an error will
occur:
```
java.lang.IllegalAccessError: class org.apache.hadoop.fs.cosn.buffer.CosNMappedBuffer (in unnamed module @0x4bb33f74) cannot access class sun.nio.ch.FileChannelImpl (in module java.base) because module java.base does not export
sun.nio.ch to unnamed module @0x4bb33f74
```

so, we should add JVM config: `--add-opens
java.base/sun.nio.ch=ALL-UNNAMED`.
@github-actions github-actions bot mentioned this pull request Apr 9, 2025
Merged
dataroaring pushed a commit that referenced this pull request Apr 11, 2025
@github-actions @wuwenchi
branch-3.0: [fix](iceberg)Fix the failure of creating a table in COS #…
c75b7e7 
…49885 (#49901)

Cherry-picked from #49885

Co-authored-by: wuwenchi <wuwenchi@selectdb.com>
@gavinchou gavinchou mentioned this pull request Apr 23, 2025
Closed
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@wuwenchi @koarz
[fix](iceberg)Fix the failure of creating a table in COS (apache#49885)
6ff2a6e 
### What problem does this PR solve?

Problem Summary:

When use JDK17 with cos, if create an iceberg table, an error will
occur:
```
java.lang.IllegalAccessError: class org.apache.hadoop.fs.cosn.buffer.CosNMappedBuffer (in unnamed module @0x4bb33f74) cannot access class sun.nio.ch.FileChannelImpl (in module java.base) because module java.base does not export
sun.nio.ch to unnamed module @0x4bb33f74
```

so, we should add JVM config: `--add-opens
java.base/sun.nio.ch=ALL-UNNAMED`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@CalvinKirs CalvinKirs CalvinKirs approved these changes

+2 more reviewers

@suxiaogang223 suxiaogang223 suxiaogang223 approved these changes

@hubgeter hubgeter hubgeter approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.5-merged p0_test reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@wuwenchi @hello-stephen @doris-robot @morningman @CalvinKirs @suxiaogang223 @hubgeter @dataroaring