[Fix](catalog)Fixes query failures for Paimon tables stored in Kerberized HDFS #47192

CalvinKirs · 2025-01-18T01:39:40Z

What problem does this PR solve?

Using JNI to directly read Paimon tables can lead to query failures when the Paimon storage is on HDFS with Kerberos authentication enabled.

Reproduction Steps:

Create a Paimon catalog stored on an HDFS cluster with Kerberos authentication enabled.
Execute the command: SET force_jni_scanner=true;.
To ensure a clean environment, restart the BE (Backend) service.
Perform any query on a table within the catalog.

2025-01-18 09:25:06  WARN Thread-13 org.apache.doris.paimon.PaimonJniScanner.open(PaimonJniScanner.java:126) - Failed to open paimon_scanner: java.io.UncheckedIOException: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
com.google.common.util.concurrent.UncheckedExecutionException: java.io.UncheckedIOException: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2085)
        at com.google.common.cache.LocalCache.get(LocalCache.java:4017)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4040)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4989)
        at org.apache.doris.paimon.PaimonTableCache.getTable(PaimonTableCache.java:84)
        at org.apache.doris.paimon.PaimonJniScanner.initTable(PaimonJniScanner.java:237)
        at org.apache.doris.paimon.PaimonJniScanner.open(PaimonJniScanner.java:122)
Caused by: java.io.UncheckedIOException: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
        at org.apache.paimon.hive.HiveCatalog.createHiveCatalog(HiveCatalog.java:708)
        at org.apache.paimon.hive.HiveCatalogFactory.create(HiveCatalogFactory.java:48)
        at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:76)
        at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:66)
        at org.apache.doris.paimon.PaimonTableCache.createCatalog(PaimonTableCache.java:75)
        at org.apache.doris.paimon.PaimonTableCache.loadTable(PaimonTableCache.java:58)
        at org.apache.doris.paimon.PaimonTableCache.access$000(PaimonTableCache.java:38)
        at org.apache.doris.paimon.PaimonTableCache$1.load(PaimonTableCache.java:51)
        at org.apache.doris.paimon.PaimonTableCache$1.load(PaimonTableCache.java:48)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3574)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2316)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2189)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2079)
        ... 6 more

changes

This PR addresses an issue where queries fail when attempting to directly read Paimon tables using JNI, specifically in environments where HDFS is used as the storage backend and Kerberos authentication is enabled. The failure is caused by the lack of proper Kerberos authentication handling in the JNI implementation.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
  After updating the code, repeat the reproduction steps outlined above.
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-01-18T01:39:44Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

CalvinKirs · 2025-01-18T01:39:50Z

run buildall

doris-robot · 2025-01-18T02:18:36Z

TPC-H: Total hot run time: 32270 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b2b35eb2a419afe78b8e012607f5aff0a382f3a7, data reload: false

------ Round 1 ----------------------------------
q1	17572	5435	5324	5324
q2	2053	302	176	176
q3	10583	1238	713	713
q4	10205	964	509	509
q5	7551	2339	2167	2167
q6	193	163	132	132
q7	896	740	602	602
q8	9227	1326	1105	1105
q9	5163	4893	4874	4874
q10	6860	2336	1897	1897
q11	476	259	252	252
q12	348	360	214	214
q13	17782	3686	3118	3118
q14	234	222	215	215
q15	518	481	473	473
q16	626	611	587	587
q17	569	864	338	338
q18	6981	6665	6612	6612
q19	3000	928	513	513
q20	303	311	193	193
q21	2665	2127	1964	1964
q22	364	329	292	292
Total cold run time: 104169 ms
Total hot run time: 32270 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5611	5470	5455	5455
q2	232	332	240	240
q3	2289	2662	2353	2353
q4	1430	1795	1366	1366
q5	4314	4739	4657	4657
q6	171	159	130	130
q7	2077	1989	1804	1804
q8	2629	2829	2686	2686
q9	7376	7249	7359	7249
q10	3028	3260	2772	2772
q11	561	516	473	473
q12	657	819	672	672
q13	3539	3900	3238	3238
q14	294	312	277	277
q15	510	485	449	449
q16	641	709	662	662
q17	1202	1725	1261	1261
q18	7568	7523	7276	7276
q19	772	1183	1068	1068
q20	2040	2027	1883	1883
q21	5726	5145	4908	4908
q22	615	628	563	563
Total cold run time: 53282 ms
Total hot run time: 51442 ms

doris-robot · 2025-01-18T02:30:40Z

TPC-DS: Total hot run time: 193964 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b2b35eb2a419afe78b8e012607f5aff0a382f3a7, data reload: false

query1	1312	960	970	960
query2	6297	2018	2019	2018
query3	11077	4742	4528	4528
query4	33315	23414	22843	22843
query5	3544	610	462	462
query6	300	200	178	178
query7	3989	482	319	319
query8	303	245	247	245
query9	9314	2543	2553	2543
query10	466	298	249	249
query11	17614	15244	15003	15003
query12	158	104	102	102
query13	1560	499	400	400
query14	9752	7075	6802	6802
query15	248	212	193	193
query16	7789	674	526	526
query17	1592	788	579	579
query18	2085	396	310	310
query19	208	196	163	163
query20	120	127	119	119
query21	204	132	109	109
query22	4651	4397	4320	4320
query23	34920	33643	33531	33531
query24	6525	2378	2288	2288
query25	500	450	391	391
query26	849	251	165	165
query27	2083	473	336	336
query28	5329	2444	2455	2444
query29	560	536	420	420
query30	215	186	168	168
query31	964	885	846	846
query32	70	64	57	57
query33	481	357	314	314
query34	739	894	509	509
query35	845	853	740	740
query36	997	1015	952	952
query37	121	93	82	82
query38	4268	4442	4194	4194
query39	1539	1415	1436	1415
query40	209	123	101	101
query41	50	53	63	53
query42	120	99	94	94
query43	527	534	501	501
query44	1317	811	803	803
query45	182	175	170	170
query46	870	1064	653	653
query47	1901	1912	1856	1856
query48	389	416	317	317
query49	725	494	410	410
query50	635	666	407	407
query51	7185	7081	7110	7081
query52	103	104	92	92
query53	228	251	184	184
query54	498	496	430	430
query55	85	77	83	77
query56	270	260	249	249
query57	1137	1158	1137	1137
query58	245	229	235	229
query59	3216	3154	3240	3154
query60	285	293	295	293
query61	141	136	198	136
query62	727	702	662	662
query63	224	185	186	185
query64	3402	1019	644	644
query65	3254	3157	3145	3145
query66	924	403	296	296
query67	15622	15672	15624	15624
query68	5054	836	519	519
query69	496	295	263	263
query70	1206	1154	1148	1148
query71	376	284	250	250
query72	5850	3957	3870	3870
query73	620	742	351	351
query74	10086	9115	8811	8811
query75	3162	3127	2698	2698
query76	3047	1165	749	749
query77	457	363	267	267
query78	9997	9883	9274	9274
query79	3098	786	587	587
query80	669	526	440	440
query81	489	270	240	240
query82	558	151	132	132
query83	170	167	165	165
query84	233	92	82	82
query85	787	425	303	303
query86	394	311	281	281
query87	4406	4436	4685	4436
query88	4969	2104	2083	2083
query89	395	318	291	291
query90	1789	191	187	187
query91	141	135	109	109
query92	64	54	53	53
query93	2587	886	525	525
query94	697	401	312	312
query95	351	262	253	253
query96	497	601	284	284
query97	2783	2838	2733	2733
query98	226	208	198	198
query99	1276	1348	1264	1264
Total cold run time: 288304 ms
Total hot run time: 193964 ms

doris-robot · 2025-01-18T02:35:59Z

ClickBench: Total hot run time: 30.91 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b2b35eb2a419afe78b8e012607f5aff0a382f3a7, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.03
query3	0.23	0.07	0.06
query4	1.63	0.10	0.11
query5	0.42	0.43	0.41
query6	1.14	0.65	0.65
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.59	0.50	0.50
query10	0.55	0.57	0.57
query11	0.14	0.10	0.11
query12	0.15	0.11	0.11
query13	0.61	0.60	0.61
query14	2.71	2.78	2.71
query15	0.90	0.83	0.82
query16	0.39	0.37	0.38
query17	0.99	1.04	1.05
query18	0.23	0.22	0.22
query19	1.97	1.86	1.97
query20	0.01	0.01	0.01
query21	15.36	0.94	0.58
query22	0.75	0.85	0.64
query23	15.25	1.43	0.60
query24	3.19	1.10	1.58
query25	0.15	0.26	0.16
query26	0.26	0.16	0.14
query27	0.06	0.05	0.06
query28	14.04	0.96	0.44
query29	12.61	3.95	3.25
query30	0.24	0.08	0.06
query31	2.84	0.62	0.39
query32	3.24	0.55	0.46
query33	3.03	3.04	3.02
query34	16.71	5.22	4.48
query35	4.64	4.51	4.54
query36	0.66	0.49	0.50
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.14
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.35 s
Total hot run time: 30.91 s

CalvinKirs · 2025-01-20T02:00:25Z

run p0

CalvinKirs · 2025-01-20T05:44:28Z

run p0

CalvinKirs · 2025-01-24T02:44:10Z

run buildall

doris-robot · 2025-01-24T03:26:26Z

TPC-H: Total hot run time: 31892 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e5271e30a26db01a77f68de1f330bafbd00ed6fa, data reload: false

------ Round 1 ----------------------------------
q1	17588	5458	5317	5317
q2	2045	295	170	170
q3	10627	1228	736	736
q4	10209	980	538	538
q5	7508	2365	2118	2118
q6	191	172	131	131
q7	894	759	604	604
q8	9232	1327	1125	1125
q9	5142	4887	4815	4815
q10	6810	2337	1888	1888
q11	499	273	260	260
q12	336	354	223	223
q13	17788	3689	3059	3059
q14	236	233	211	211
q15	522	475	472	472
q16	639	616	589	589
q17	568	848	319	319
q18	6851	6402	6310	6310
q19	2023	948	534	534
q20	304	303	185	185
q21	2958	2269	1983	1983
q22	361	330	305	305
Total cold run time: 103331 ms
Total hot run time: 31892 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5545	5495	5465	5465
q2	238	338	235	235
q3	2284	2659	2332	2332
q4	1372	1828	1351	1351
q5	4291	4751	4627	4627
q6	163	157	124	124
q7	1996	1957	1820	1820
q8	2625	2762	2700	2700
q9	7257	7234	7264	7234
q10	3047	3290	2780	2780
q11	578	517	505	505
q12	651	715	570	570
q13	3462	3972	3250	3250
q14	278	294	267	267
q15	522	476	477	476
q16	648	703	653	653
q17	1215	1739	1288	1288
q18	7564	7665	7143	7143
q19	793	1125	1112	1112
q20	1987	2035	1913	1913
q21	5749	5244	4862	4862
q22	622	574	558	558
Total cold run time: 52887 ms
Total hot run time: 51265 ms

doris-robot · 2025-01-24T03:38:58Z

TPC-DS: Total hot run time: 192854 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e5271e30a26db01a77f68de1f330bafbd00ed6fa, data reload: false

query1	1287	940	951	940
query2	6392	2045	2004	2004
query3	10946	4524	4543	4524
query4	60729	29070	23237	23237
query5	5496	611	499	499
query6	431	194	187	187
query7	5572	517	303	303
query8	336	246	238	238
query9	8058	2577	2579	2577
query10	459	316	258	258
query11	17587	15723	15503	15503
query12	166	113	113	113
query13	1487	581	431	431
query14	11776	7594	7278	7278
query15	216	206	191	191
query16	6867	685	489	489
query17	1130	776	598	598
query18	1376	406	315	315
query19	206	195	174	174
query20	119	117	112	112
query21	218	128	108	108
query22	4502	4733	4306	4306
query23	34740	33289	33385	33289
query24	5563	2344	2336	2336
query25	475	463	416	416
query26	683	256	161	161
query27	1878	517	343	343
query28	4097	2501	2449	2449
query29	549	563	452	452
query30	209	191	163	163
query31	944	913	832	832
query32	71	61	61	61
query33	427	377	308	308
query34	774	889	520	520
query35	829	864	797	797
query36	1020	1041	1000	1000
query37	135	109	99	99
query38	4357	4376	4251	4251
query39	1522	1455	1474	1455
query40	204	121	104	104
query41	58	58	53	53
query42	118	104	106	104
query43	526	522	493	493
query44	1360	823	837	823
query45	179	171	171	171
query46	905	1069	670	670
query47	1939	1904	1806	1806
query48	410	412	322	322
query49	733	499	429	429
query50	691	690	414	414
query51	4324	4393	4295	4295
query52	111	106	96	96
query53	242	273	199	199
query54	504	517	437	437
query55	85	86	78	78
query56	274	286	257	257
query57	1163	1214	1146	1146
query58	242	248	235	235
query59	3209	3351	3131	3131
query60	282	274	260	260
query61	121	116	112	112
query62	752	732	691	691
query63	260	202	205	202
query64	1715	1055	656	656
query65	3231	3138	3146	3138
query66	703	403	288	288
query67	16064	15718	15507	15507
query68	5039	850	532	532
query69	484	297	260	260
query70	1251	1169	1098	1098
query71	415	292	260	260
query72	6328	3895	3885	3885
query73	831	770	373	373
query74	10509	8816	8931	8816
query75	3230	3168	2696	2696
query76	3805	1177	771	771
query77	469	398	281	281
query78	10038	10033	9414	9414
query79	2841	829	591	591
query80	1562	560	440	440
query81	553	267	235	235
query82	354	147	125	125
query83	253	169	149	149
query84	301	98	69	69
query85	762	348	298	298
query86	444	294	302	294
query87	4478	4620	4419	4419
query88	3728	2217	2189	2189
query89	451	329	287	287
query90	1610	195	189	189
query91	131	138	109	109
query92	67	61	53	53
query93	2602	884	530	530
query94	734	370	314	314
query95	328	278	327	278
query96	487	633	291	291
query97	2856	2849	2735	2735
query98	213	200	197	197
query99	1328	1375	1242	1242
Total cold run time: 312516 ms
Total hot run time: 192854 ms

doris-robot · 2025-01-24T03:44:16Z

ClickBench: Total hot run time: 30.62 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e5271e30a26db01a77f68de1f330bafbd00ed6fa, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.24	0.07	0.06
query4	1.63	0.11	0.11
query5	0.41	0.41	0.40
query6	1.13	0.67	0.66
query7	0.03	0.02	0.02
query8	0.05	0.03	0.03
query9	0.60	0.51	0.49
query10	0.56	0.57	0.56
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.58	0.59
query14	2.83	2.73	2.78
query15	0.88	0.83	0.82
query16	0.39	0.38	0.38
query17	1.03	1.00	1.00
query18	0.22	0.22	0.20
query19	1.92	1.80	1.98
query20	0.01	0.01	0.01
query21	15.35	0.96	0.57
query22	0.76	0.75	0.62
query23	15.36	1.43	0.61
query24	3.38	1.34	1.16
query25	0.18	0.14	0.06
query26	0.22	0.14	0.13
query27	0.07	0.05	0.04
query28	14.16	0.93	0.43
query29	12.59	3.84	3.22
query30	0.25	0.09	0.06
query31	2.82	0.60	0.38
query32	3.22	0.54	0.46
query33	2.97	2.97	2.98
query34	16.46	5.13	4.48
query35	4.48	4.56	4.50
query36	0.63	0.49	0.50
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.04
Total cold run time: 106.33 s
Total hot run time: 30.62 s

CalvinKirs · 2025-01-24T06:32:50Z

run p0

CalvinKirs · 2025-01-24T06:32:58Z

run external

CalvinKirs · 2025-01-26T07:13:39Z

run external

CalvinKirs · 2025-01-26T14:47:26Z

run buildall

doris-robot · 2025-01-26T15:06:19Z

TPC-H: Total hot run time: 32413 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 895c190276454ade68806ffed22fc0f893a622c8, data reload: false

------ Round 1 ----------------------------------
q1	17586	5563	5368	5368
q2	2045	298	170	170
q3	10423	1278	725	725
q4	10230	956	530	530
q5	7540	2459	2162	2162
q6	201	168	136	136
q7	906	762	630	630
q8	9269	1368	1151	1151
q9	5404	4928	5020	4928
q10	6844	2345	1895	1895
q11	484	271	271	271
q12	343	359	225	225
q13	17788	3670	3095	3095
q14	231	238	207	207
q15	536	471	484	471
q16	638	638	599	599
q17	565	879	334	334
q18	7079	6599	6438	6438
q19	1236	955	555	555
q20	321	327	194	194
q21	3049	2251	2021	2021
q22	367	335	308	308
Total cold run time: 103085 ms
Total hot run time: 32413 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5523	5513	5479	5479
q2	241	333	232	232
q3	2276	2646	2318	2318
q4	1418	1784	1373	1373
q5	4349	4727	4702	4702
q6	166	158	125	125
q7	2103	1966	1890	1890
q8	2673	2813	2695	2695
q9	7231	7224	7165	7165
q10	2965	3279	2765	2765
q11	601	533	494	494
q12	650	746	621	621
q13	3571	3938	3311	3311
q14	282	287	272	272
q15	527	471	465	465
q16	652	686	641	641
q17	1235	1754	1282	1282
q18	7608	7464	7356	7356
q19	823	1127	1141	1127
q20	1991	2024	1887	1887
q21	5699	5193	4989	4989
q22	596	591	543	543
Total cold run time: 53180 ms
Total hot run time: 51732 ms

doris-robot · 2025-01-26T15:17:47Z

TPC-DS: Total hot run time: 184510 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 895c190276454ade68806ffed22fc0f893a622c8, data reload: false

query1	984	381	390	381
query2	6525	2075	2003	2003
query3	6787	223	225	223
query4	33411	23341	23041	23041
query5	4301	646	483	483
query6	282	197	186	186
query7	4613	503	307	307
query8	301	235	220	220
query9	9712	2692	2697	2692
query10	473	313	255	255
query11	18336	15372	14969	14969
query12	157	105	105	105
query13	1656	541	408	408
query14	10919	6780	6703	6703
query15	250	182	178	178
query16	8009	606	419	419
query17	1580	741	567	567
query18	2103	395	292	292
query19	218	185	148	148
query20	114	108	111	108
query21	206	120	98	98
query22	4313	4343	4134	4134
query23	33639	32684	33023	32684
query24	6464	2254	2279	2254
query25	518	439	385	385
query26	777	277	152	152
query27	2019	475	334	334
query28	5164	2490	2442	2442
query29	683	531	437	437
query30	232	182	158	158
query31	975	873	810	810
query32	94	62	62	62
query33	503	348	294	294
query34	764	860	494	494
query35	800	807	727	727
query36	979	1015	971	971
query37	127	111	78	78
query38	4200	4128	4067	4067
query39	1442	1413	1413	1413
query40	205	112	101	101
query41	53	55	51	51
query42	123	107	109	107
query43	532	525	493	493
query44	1353	844	821	821
query45	181	170	161	161
query46	865	1024	638	638
query47	1782	1798	1750	1750
query48	384	404	327	327
query49	745	489	399	399
query50	627	666	388	388
query51	4231	4170	4070	4070
query52	105	108	89	89
query53	237	261	188	188
query54	488	483	408	408
query55	82	85	84	84
query56	268	254	244	244
query57	1142	1141	1084	1084
query58	240	229	241	229
query59	2927	3212	2961	2961
query60	282	276	275	275
query61	120	116	111	111
query62	772	727	637	637
query63	232	192	192	192
query64	3837	984	645	645
query65	3267	3138	3172	3138
query66	865	401	325	325
query67	15693	15455	15515	15455
query68	2181	836	583	583
query69	405	300	265	265
query70	1174	1181	1152	1152
query71	325	282	286	282
query72	5006	4061	3822	3822
query73	638	756	364	364
query74	9881	8887	8850	8850
query75	3177	3173	2656	2656
query76	2202	1198	794	794
query77	333	363	294	294
query78	10248	9973	9298	9298
query79	1013	888	604	604
query80	609	582	442	442
query81	468	278	242	242
query82	1097	160	119	119
query83	275	183	154	154
query84	236	86	92	86
query85	727	364	304	304
query86	344	321	305	305
query87	4522	4524	4389	4389
query88	3233	2188	2167	2167
query89	392	339	301	301
query90	1679	194	188	188
query91	136	139	107	107
query92	63	63	54	54
query93	974	851	541	541
query94	492	401	293	293
query95	335	269	266	266
query96	505	615	283	283
query97	2781	2894	2787	2787
query98	214	197	203	197
query99	1281	1362	1268	1268
Total cold run time: 273214 ms
Total hot run time: 184510 ms

doris-robot · 2025-01-26T15:23:05Z

ClickBench: Total hot run time: 31.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 895c190276454ade68806ffed22fc0f893a622c8, data reload: false

query1	0.04	0.04	0.05
query2	0.08	0.03	0.04
query3	0.24	0.06	0.07
query4	1.62	0.10	0.10
query5	0.40	0.42	0.41
query6	1.19	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.60	0.49	0.50
query10	0.55	0.55	0.57
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.59	0.60
query14	2.84	2.86	2.86
query15	0.91	0.83	0.83
query16	0.41	0.38	0.38
query17	1.03	1.04	1.06
query18	0.21	0.20	0.21
query19	1.92	1.82	2.00
query20	0.01	0.01	0.01
query21	15.36	0.93	0.57
query22	0.76	0.97	0.78
query23	14.99	1.40	0.54
query24	2.69	1.24	1.66
query25	0.16	0.18	0.04
query26	0.21	0.14	0.14
query27	0.06	0.06	0.05
query28	14.26	0.98	0.43
query29	12.57	4.02	3.29
query30	0.25	0.09	0.07
query31	2.82	0.60	0.40
query32	3.24	0.55	0.46
query33	2.95	3.06	3.01
query34	16.70	5.16	4.52
query35	4.54	4.50	4.46
query36	0.66	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 105.7 s
Total hot run time: 31.11 s

CalvinKirs · 2025-01-27T00:10:16Z

run buildall

doris-robot · 2025-01-27T00:48:21Z

TPC-H: Total hot run time: 32330 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 895c190276454ade68806ffed22fc0f893a622c8, data reload: false

------ Round 1 ----------------------------------
q1	17585	5548	5400	5400
q2	2040	301	165	165
q3	10429	1238	742	742
q4	10221	979	543	543
q5	7550	2371	2144	2144
q6	198	163	130	130
q7	911	768	600	600
q8	9233	1345	1164	1164
q9	5013	4909	4885	4885
q10	6895	2330	1875	1875
q11	485	275	255	255
q12	345	360	223	223
q13	17782	3667	3119	3119
q14	225	243	204	204
q15	515	473	474	473
q16	619	625	584	584
q17	572	848	330	330
q18	6907	6395	6356	6356
q19	2472	953	542	542
q20	308	322	191	191
q21	2835	2285	2086	2086
q22	390	352	319	319
Total cold run time: 103530 ms
Total hot run time: 32330 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5520	5352	5420	5352
q2	246	341	238	238
q3	2373	2775	2296	2296
q4	1400	1786	1375	1375
q5	4292	4749	4704	4704
q6	174	158	126	126
q7	2048	1922	1763	1763
q8	2617	2830	2734	2734
q9	7317	7161	7218	7161
q10	3061	3308	2823	2823
q11	580	539	504	504
q12	655	767	574	574
q13	3506	3879	3327	3327
q14	292	316	272	272
q15	501	468	478	468
q16	662	699	661	661
q17	1256	1745	1294	1294
q18	7752	7288	7372	7288
q19	799	1153	1101	1101
q20	1956	2014	1896	1896
q21	5862	5204	4961	4961
q22	590	631	612	612
Total cold run time: 53459 ms
Total hot run time: 51530 ms

doris-robot · 2025-01-27T01:00:43Z

TPC-DS: Total hot run time: 191121 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 895c190276454ade68806ffed22fc0f893a622c8, data reload: false

query1	1281	927	943	927
query2	6218	2061	2017	2017
query3	11125	4543	4682	4543
query4	61207	28906	23000	23000
query5	5490	605	450	450
query6	442	195	193	193
query7	5540	502	300	300
query8	346	246	232	232
query9	8151	2704	2704	2704
query10	451	309	257	257
query11	17989	15191	15400	15191
query12	166	109	111	109
query13	1400	545	419	419
query14	11083	7469	6558	6558
query15	211	212	186	186
query16	7233	655	514	514
query17	1129	756	615	615
query18	1919	409	333	333
query19	210	186	167	167
query20	124	117	115	115
query21	210	134	107	107
query22	4408	4620	4640	4620
query23	33811	33202	33481	33202
query24	5501	2351	2329	2329
query25	466	454	423	423
query26	646	274	158	158
query27	1750	492	348	348
query28	4056	2482	2473	2473
query29	550	553	427	427
query30	233	190	162	162
query31	934	869	831	831
query32	71	60	58	58
query33	440	357	296	296
query34	750	879	520	520
query35	821	851	770	770
query36	1025	1062	954	954
query37	139	103	80	80
query38	4292	4356	4340	4340
query39	1471	1452	1443	1443
query40	216	113	102	102
query41	50	48	47	47
query42	126	105	104	104
query43	520	534	490	490
query44	1348	835	858	835
query45	179	174	168	168
query46	883	1066	675	675
query47	1882	1937	1841	1841
query48	393	404	312	312
query49	731	482	393	393
query50	657	698	406	406
query51	4235	4260	4233	4233
query52	102	100	95	95
query53	238	264	198	198
query54	484	507	420	420
query55	85	82	88	82
query56	260	261	260	260
query57	1150	1216	1129	1129
query58	254	270	265	265
query59	3192	3085	3121	3085
query60	291	269	257	257
query61	114	115	118	115
query62	783	755	677	677
query63	234	191	201	191
query64	1332	1030	666	666
query65	3248	3203	3172	3172
query66	714	386	303	303
query67	15885	15505	15186	15186
query68	3638	802	532	532
query69	478	297	262	262
query70	1212	1149	1108	1108
query71	407	281	258	258
query72	6105	3829	3893	3829
query73	666	753	362	362
query74	9812	9236	8679	8679
query75	3201	3145	2677	2677
query76	2998	1142	768	768
query77	481	356	269	269
query78	10001	10006	9233	9233
query79	2283	817	616	616
query80	684	522	444	444
query81	505	276	247	247
query82	240	163	128	128
query83	173	170	152	152
query84	304	113	74	74
query85	738	373	291	291
query86	372	313	301	301
query87	4456	4480	4393	4393
query88	3665	2195	2194	2194
query89	397	325	290	290
query90	1717	190	192	190
query91	148	138	107	107
query92	67	137	52	52
query93	2358	854	540	540
query94	671	404	302	302
query95	340	270	260	260
query96	497	613	285	285
query97	2787	2859	2724	2724
query98	228	192	193	192
query99	1294	1360	1284	1284
Total cold run time: 305798 ms
Total hot run time: 191121 ms

doris-robot · 2025-01-27T01:06:00Z

ClickBench: Total hot run time: 30.4 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 895c190276454ade68806ffed22fc0f893a622c8, data reload: false

query1	0.03	0.06	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.62	0.10	0.10
query5	0.42	0.42	0.40
query6	1.14	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.58	0.49	0.54
query10	0.55	0.57	0.55
query11	0.16	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.59	0.61
query14	2.72	2.89	2.85
query15	0.89	0.83	0.83
query16	0.38	0.38	0.38
query17	1.07	1.07	1.04
query18	0.22	0.22	0.20
query19	1.85	1.90	2.00
query20	0.01	0.01	0.01
query21	15.37	0.97	0.59
query22	0.74	0.88	0.68
query23	15.22	1.45	0.54
query24	3.19	0.85	0.49
query25	0.15	0.18	0.08
query26	0.39	0.16	0.14
query27	0.05	0.07	0.04
query28	13.42	1.05	0.44
query29	12.58	3.97	3.28
query30	0.25	0.09	0.06
query31	2.81	0.57	0.38
query32	3.22	0.55	0.46
query33	3.00	3.06	3.03
query34	16.80	5.14	4.49
query35	4.54	4.57	4.52
query36	0.65	0.49	0.50
query37	0.10	0.07	0.06
query38	0.06	0.03	0.03
query39	0.04	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 105.65 s
Total hot run time: 30.4 s

CalvinKirs · 2025-01-27T01:55:43Z

run buildall

doris-robot · 2025-01-27T02:33:41Z

TPC-H: Total hot run time: 32214 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 90360da67e4353614d47b5e3cddcd8f726d99108, data reload: false

------ Round 1 ----------------------------------
q1	17571	5446	5307	5307
q2	2044	303	165	165
q3	10426	1229	723	723
q4	10206	970	538	538
q5	7523	2349	2156	2156
q6	189	164	129	129
q7	885	775	610	610
q8	9230	1341	1141	1141
q9	5124	4857	4851	4851
q10	6816	2304	1901	1901
q11	456	277	250	250
q12	344	354	226	226
q13	18060	3724	3129	3129
q14	219	233	199	199
q15	507	474	473	473
q16	614	626	589	589
q17	560	865	325	325
q18	6912	6555	6436	6436
q19	1547	949	539	539
q20	302	313	189	189
q21	2864	2273	2029	2029
q22	363	345	309	309
Total cold run time: 102762 ms
Total hot run time: 32214 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5528	5518	5559	5518
q2	235	328	230	230
q3	2279	2650	2302	2302
q4	1417	1833	1388	1388
q5	4372	4855	4788	4788
q6	168	160	130	130
q7	2114	1978	1864	1864
q8	2668	2908	2747	2747
q9	7427	7282	7328	7282
q10	3103	3376	2895	2895
q11	615	534	522	522
q12	686	796	611	611
q13	3766	4276	3611	3611
q14	308	317	307	307
q15	535	495	472	472
q16	682	714	644	644
q17	1272	1758	1316	1316
q18	7737	7376	7493	7376
q19	815	985	1150	985
q20	2032	2076	1920	1920
q21	5712	5307	4774	4774
q22	612	579	592	579
Total cold run time: 54083 ms
Total hot run time: 52261 ms

doris-robot · 2025-01-27T02:46:09Z

TPC-DS: Total hot run time: 191766 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 90360da67e4353614d47b5e3cddcd8f726d99108, data reload: false

query1	1290	967	948	948
query2	6319	2006	1999	1999
query3	10937	4599	4374	4374
query4	61026	29480	23059	23059
query5	5445	598	455	455
query6	427	196	184	184
query7	5534	503	318	318
query8	346	235	232	232
query9	8258	2702	2672	2672
query10	444	309	248	248
query11	17676	15074	15503	15074
query12	160	108	108	108
query13	1394	569	431	431
query14	11569	7008	7127	7008
query15	217	210	185	185
query16	7275	659	507	507
query17	1124	730	597	597
query18	1939	418	334	334
query19	207	185	158	158
query20	116	115	118	115
query21	218	129	104	104
query22	4460	4802	4287	4287
query23	33911	33598	33501	33501
query24	5483	2430	2298	2298
query25	470	466	404	404
query26	635	274	152	152
query27	1776	463	339	339
query28	4854	2486	2481	2481
query29	518	565	416	416
query30	215	190	153	153
query31	919	879	815	815
query32	65	58	56	56
query33	440	360	323	323
query34	761	869	527	527
query35	854	853	779	779
query36	1008	1043	946	946
query37	127	96	76	76
query38	4245	4385	4130	4130
query39	1502	1433	1445	1433
query40	211	113	102	102
query41	50	46	48	46
query42	118	98	101	98
query43	522	529	493	493
query44	1365	830	854	830
query45	180	177	169	169
query46	876	1037	656	656
query47	1908	1915	1856	1856
query48	404	416	320	320
query49	718	490	390	390
query50	649	696	405	405
query51	4313	4343	4238	4238
query52	115	105	93	93
query53	237	256	192	192
query54	483	505	419	419
query55	87	78	75	75
query56	252	273	258	258
query57	1191	1215	1127	1127
query58	248	236	251	236
query59	3104	3228	3228	3228
query60	273	283	260	260
query61	138	118	117	117
query62	734	696	660	660
query63	222	187	182	182
query64	1294	1016	714	714
query65	3267	3164	3163	3163
query66	732	394	321	321
query67	15966	15737	15493	15493
query68	4798	776	538	538
query69	482	295	270	270
query70	1192	1166	1104	1104
query71	401	284	246	246
query72	6028	3971	3860	3860
query73	754	776	356	356
query74	10082	8896	9001	8896
query75	3210	3221	2634	2634
query76	3562	1191	773	773
query77	511	361	285	285
query78	10016	9982	9321	9321
query79	2824	795	600	600
query80	673	524	469	469
query81	504	286	243	243
query82	306	144	127	127
query83	177	177	151	151
query84	293	88	75	75
query85	735	350	306	306
query86	386	306	307	306
query87	4471	4544	4525	4525
query88	4237	2159	2154	2154
query89	405	329	280	280
query90	1685	189	193	189
query91	135	134	113	113
query92	62	60	50	50
query93	2734	886	533	533
query94	677	389	306	306
query95	330	307	251	251
query96	496	613	282	282
query97	2831	2855	2714	2714
query98	227	196	190	190
query99	1271	1404	1252	1252
Total cold run time: 310316 ms
Total hot run time: 191766 ms

doris-robot · 2025-01-27T02:51:27Z

ClickBench: Total hot run time: 30.57 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 90360da67e4353614d47b5e3cddcd8f726d99108, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.05	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.43	0.42	0.41
query6	1.14	0.68	0.65
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.51	0.50
query10	0.56	0.57	0.56
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.60	0.59	0.59
query14	2.73	2.82	2.73
query15	0.89	0.83	0.82
query16	0.37	0.39	0.38
query17	1.03	1.08	1.03
query18	0.22	0.19	0.21
query19	1.84	1.82	2.02
query20	0.02	0.01	0.01
query21	15.38	0.93	0.60
query22	0.75	0.71	0.66
query23	15.35	1.46	0.59
query24	3.00	0.87	1.74
query25	0.11	0.20	0.06
query26	0.43	0.14	0.13
query27	0.04	0.05	0.04
query28	13.96	1.03	0.44
query29	12.58	3.89	3.27
query30	0.25	0.10	0.06
query31	2.83	0.59	0.40
query32	3.23	0.54	0.47
query33	2.95	3.00	3.01
query34	16.58	5.14	4.52
query35	4.51	4.49	4.50
query36	0.64	0.49	0.49
query37	0.10	0.06	0.05
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.14	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.82 s
Total hot run time: 30.57 s

CalvinKirs · 2025-02-05T09:26:05Z

run buildall

zy-kkk

LGTM

github-actions · 2025-02-10T08:11:46Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-02-10T08:11:48Z

PR approved by anyone and no changes requested.

…ized HDFS (#47192) ### What problem does this PR solve? Using JNI to directly read Paimon tables can lead to query failures when the Paimon storage is on HDFS with Kerberos authentication enabled. #### Reproduction Steps: - Create a Paimon catalog stored on an HDFS cluster with Kerberos authentication enabled. - Execute the command: SET force_jni_scanner=true;. - To ensure a clean environment, restart the BE (Backend) service. - Perform any query on a table within the catalog. ``` 2025-01-18 09:25:06 WARN Thread-13 org.apache.doris.paimon.PaimonJniScanner.open(PaimonJniScanner.java:126) - Failed to open paimon_scanner: java.io.UncheckedIOException: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] com.google.common.util.concurrent.UncheckedExecutionException: java.io.UncheckedIOException: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2085) at com.google.common.cache.LocalCache.get(LocalCache.java:4017) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4040) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4989) at org.apache.doris.paimon.PaimonTableCache.getTable(PaimonTableCache.java:84) at org.apache.doris.paimon.PaimonJniScanner.initTable(PaimonJniScanner.java:237) at org.apache.doris.paimon.PaimonJniScanner.open(PaimonJniScanner.java:122) Caused by: java.io.UncheckedIOException: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] at org.apache.paimon.hive.HiveCatalog.createHiveCatalog(HiveCatalog.java:708) at org.apache.paimon.hive.HiveCatalogFactory.create(HiveCatalogFactory.java:48) at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:76) at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:66) at org.apache.doris.paimon.PaimonTableCache.createCatalog(PaimonTableCache.java:75) at org.apache.doris.paimon.PaimonTableCache.loadTable(PaimonTableCache.java:58) at org.apache.doris.paimon.PaimonTableCache.access$000(PaimonTableCache.java:38) at org.apache.doris.paimon.PaimonTableCache$1.load(PaimonTableCache.java:51) at org.apache.doris.paimon.PaimonTableCache$1.load(PaimonTableCache.java:48) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3574) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2316) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2189) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2079) ... 6 more ``` #### changes This PR addresses an issue where queries fail when attempting to directly read Paimon tables using JNI, specifically in environments where HDFS is used as the storage backend and Kerberos authentication is enabled. The failure is caused by the lack of proper Kerberos authentication handling in the JNI implementation.

add more hudi jni test cases. These tests depends on apache#47192 and part of apache#47299(the jni-util part)

…ed in Kerberized HDFS #47192 (#47694) Cherry-picked from #47192 Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>

### What problem does this PR solve? add more hudi jni test cases. These tests depends on apache#47192 and part of apache#47299(the jni-util part)

…ized HDFS (apache#47192) ### What problem does this PR solve? Using JNI to directly read Paimon tables can lead to query failures when the Paimon storage is on HDFS with Kerberos authentication enabled. #### Reproduction Steps: - Create a Paimon catalog stored on an HDFS cluster with Kerberos authentication enabled. - Execute the command: SET force_jni_scanner=true;. - To ensure a clean environment, restart the BE (Backend) service. - Perform any query on a table within the catalog. ``` 2025-01-18 09:25:06 WARN Thread-13 org.apache.doris.paimon.PaimonJniScanner.open(PaimonJniScanner.java:126) - Failed to open paimon_scanner: java.io.UncheckedIOException: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] com.google.common.util.concurrent.UncheckedExecutionException: java.io.UncheckedIOException: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2085) at com.google.common.cache.LocalCache.get(LocalCache.java:4017) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4040) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4989) at org.apache.doris.paimon.PaimonTableCache.getTable(PaimonTableCache.java:84) at org.apache.doris.paimon.PaimonJniScanner.initTable(PaimonJniScanner.java:237) at org.apache.doris.paimon.PaimonJniScanner.open(PaimonJniScanner.java:122) Caused by: java.io.UncheckedIOException: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] at org.apache.paimon.hive.HiveCatalog.createHiveCatalog(HiveCatalog.java:708) at org.apache.paimon.hive.HiveCatalogFactory.create(HiveCatalogFactory.java:48) at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:76) at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:66) at org.apache.doris.paimon.PaimonTableCache.createCatalog(PaimonTableCache.java:75) at org.apache.doris.paimon.PaimonTableCache.loadTable(PaimonTableCache.java:58) at org.apache.doris.paimon.PaimonTableCache.access$000(PaimonTableCache.java:38) at org.apache.doris.paimon.PaimonTableCache$1.load(PaimonTableCache.java:51) at org.apache.doris.paimon.PaimonTableCache$1.load(PaimonTableCache.java:48) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3574) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2316) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2189) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2079) ... 6 more ``` #### changes This PR addresses an issue where queries fail when attempting to directly read Paimon tables using JNI, specifically in environments where HDFS is used as the storage backend and Kerberos authentication is enabled. The failure is caused by the lack of proper Kerberos authentication handling in the JNI implementation.

### What problem does this PR solve? add more hudi jni test cases. These tests depends on apache#47192 and part of apache#47299(the jni-util part)

…ed in Kerberized HDFS #47192 (#47693) Cherry-picked from #47192 Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>

### What problem does this PR solve? add more hudi jni test cases. These tests depends on #47192 and part of #47299(the jni-util part)

### What problem does this PR solve? add more hudi jni test cases. These tests depends on apache#47192 and part of apache#47299(the jni-util part)

…ized HDFS (apache#47192) ### What problem does this PR solve? Using JNI to directly read Paimon tables can lead to query failures when the Paimon storage is on HDFS with Kerberos authentication enabled. #### Reproduction Steps: - Create a Paimon catalog stored on an HDFS cluster with Kerberos authentication enabled. - Execute the command: SET force_jni_scanner=true;. - To ensure a clean environment, restart the BE (Backend) service. - Perform any query on a table within the catalog. ``` 2025-01-18 09:25:06 WARN Thread-13 org.apache.doris.paimon.PaimonJniScanner.open(PaimonJniScanner.java:126) - Failed to open paimon_scanner: java.io.UncheckedIOException: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] com.google.common.util.concurrent.UncheckedExecutionException: java.io.UncheckedIOException: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2085) at com.google.common.cache.LocalCache.get(LocalCache.java:4017) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4040) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4989) at org.apache.doris.paimon.PaimonTableCache.getTable(PaimonTableCache.java:84) at org.apache.doris.paimon.PaimonJniScanner.initTable(PaimonJniScanner.java:237) at org.apache.doris.paimon.PaimonJniScanner.open(PaimonJniScanner.java:122) Caused by: java.io.UncheckedIOException: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] at org.apache.paimon.hive.HiveCatalog.createHiveCatalog(HiveCatalog.java:708) at org.apache.paimon.hive.HiveCatalogFactory.create(HiveCatalogFactory.java:48) at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:76) at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:66) at org.apache.doris.paimon.PaimonTableCache.createCatalog(PaimonTableCache.java:75) at org.apache.doris.paimon.PaimonTableCache.loadTable(PaimonTableCache.java:58) at org.apache.doris.paimon.PaimonTableCache.access$000(PaimonTableCache.java:38) at org.apache.doris.paimon.PaimonTableCache$1.load(PaimonTableCache.java:51) at org.apache.doris.paimon.PaimonTableCache$1.load(PaimonTableCache.java:48) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3574) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2316) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2189) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2079) ... 6 more ``` #### changes This PR addresses an issue where queries fail when attempting to directly read Paimon tables using JNI, specifically in environments where HDFS is used as the storage backend and Kerberos authentication is enabled. The failure is caused by the lack of proper Kerberos authentication handling in the JNI implementation.

CalvinKirs requested a review from morningman as a code owner January 18, 2025 01:39

CalvinKirs added dev/2.1.x dev/3.0.x labels Jan 18, 2025

zy-kkk approved these changes Feb 10, 2025

View reviewed changes

hubgeter approved these changes Feb 10, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 10, 2025

github-actions bot added the reviewed label Feb 10, 2025

CalvinKirs merged commit e5f4dea into apache:master Feb 10, 2025
26 of 28 checks passed

CalvinKirs deleted the master-fix-paimon-jni branch February 10, 2025 08:15

github-actions bot mentioned this pull request Feb 10, 2025

branch-3.0: [Fix](catalog)Fixes query failures for Paimon tables stored in Kerberized HDFS #47192 #47693

Merged

github-actions bot mentioned this pull request Feb 10, 2025

branch-2.1: [Fix](catalog)Fixes query failures for Paimon tables stored in Kerberized HDFS #47192 #47694

Merged

suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Feb 11, 2025

[test](hudi) add more hudi jni test cases (apache#47431)

14a0a8f

add more hudi jni test cases. These tests depends on apache#47192 and part of apache#47299(the jni-util part)

suxiaogang223 mentioned this pull request Feb 11, 2025

[charry-pick](branch-2.1) add more hudi jni test cases #47431 #47764

Merged

16 tasks

morningman pushed a commit that referenced this pull request Feb 12, 2025

branch-2.1: [Fix](catalog)Fixes query failures for Paimon tables stor…

b4af671

…ed in Kerberized HDFS #47192 (#47694) Cherry-picked from #47192 Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>

morningman added dev/2.1.9-merged and removed dev/2.1.x labels Feb 12, 2025

morningman added the usercase Important user case type label label Feb 24, 2025

suxiaogang223 mentioned this pull request Feb 25, 2025

[cherry-pick](branch-3.0) add more hudi jni test cases (#47431) #48311

Merged

16 tasks

dataroaring pushed a commit that referenced this pull request Feb 25, 2025

branch-3.0: [Fix](catalog)Fixes query failures for Paimon tables stor…

704047c

…ed in Kerberized HDFS #47192 (#47693) Cherry-picked from #47192 Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>

dataroaring added dev/3.0.5-merged and removed dev/3.0.x labels Feb 25, 2025

dataroaring pushed a commit that referenced this pull request Feb 26, 2025

[cherry-pick](branch-3.0) add more hudi jni test cases (#47431) (#48311)

c88f97e

### What problem does this PR solve? add more hudi jni test cases. These tests depends on #47192 and part of #47299(the jni-util part)

yiguolei mentioned this pull request Mar 25, 2025

2.1.9 Release Note #49443

Closed

gavinchou mentioned this pull request Apr 23, 2025

Release Notes 3.0.5 #50333

Closed

[Fix](catalog)Fixes query failures for Paimon tables stored in Kerberized HDFS #47192

[Fix](catalog)Fixes query failures for Paimon tables stored in Kerberized HDFS #47192

Uh oh!

Conversation

CalvinKirs commented Jan 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Reproduction Steps:

changes

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jan 18, 2025

Uh oh!

CalvinKirs commented Jan 18, 2025

Uh oh!

doris-robot commented Jan 18, 2025

Uh oh!

doris-robot commented Jan 18, 2025

Uh oh!

doris-robot commented Jan 18, 2025

Uh oh!

CalvinKirs commented Jan 20, 2025

Uh oh!

CalvinKirs commented Jan 20, 2025

Uh oh!

CalvinKirs commented Jan 24, 2025

Uh oh!

doris-robot commented Jan 24, 2025

Uh oh!

doris-robot commented Jan 24, 2025

Uh oh!

doris-robot commented Jan 24, 2025

Uh oh!

CalvinKirs commented Jan 24, 2025

Uh oh!

CalvinKirs commented Jan 24, 2025

Uh oh!

CalvinKirs commented Jan 26, 2025

Uh oh!

CalvinKirs commented Jan 26, 2025

Uh oh!

doris-robot commented Jan 26, 2025

Uh oh!

doris-robot commented Jan 26, 2025

Uh oh!

doris-robot commented Jan 26, 2025

Uh oh!

CalvinKirs commented Jan 27, 2025

Uh oh!

doris-robot commented Jan 27, 2025

Uh oh!

doris-robot commented Jan 27, 2025

Uh oh!

doris-robot commented Jan 27, 2025

Uh oh!

CalvinKirs commented Jan 27, 2025

Uh oh!

doris-robot commented Jan 27, 2025

Uh oh!

doris-robot commented Jan 27, 2025

Uh oh!

doris-robot commented Jan 27, 2025

Uh oh!

CalvinKirs commented Feb 5, 2025

Uh oh!

zy-kkk left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 10, 2025

Uh oh!

github-actions bot commented Feb 10, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

CalvinKirs commented Jan 18, 2025 •

edited

Loading