Skip to content

Conversation

@suxiaogang223
Copy link
Contributor

What problem does this PR solve?

Related PR: #44267

Problem Summary:
If we set hudi_jni_scanner to an incorrect value, jni_connector will be null, causing a core dump.
So we set hadoop as default hudi jni scanner.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Mar 4, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223
Copy link
Contributor Author

run buildall

@suxiaogang223 suxiaogang223 changed the title [fix](hudi) set hadoop hudi jni reader as default [fix](hudi) Set Hadoop Hudi Jni reader as default Mar 4, 2025
@doris-robot
Copy link

TPC-H: Total hot run time: 31836 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bf11101155b91ddddb9b562016773c6ac0e2c7af, data reload: false

------ Round 1 ----------------------------------
q1	17638	5250	5092	5092
q2	2062	316	186	186
q3	10515	1280	741	741
q4	10243	1061	558	558
q5	7658	2381	2384	2381
q6	198	170	133	133
q7	937	781	608	608
q8	9315	1336	1145	1145
q9	4976	4709	4794	4709
q10	6843	2299	1889	1889
q11	482	287	262	262
q12	359	362	229	229
q13	17765	3765	3086	3086
q14	235	227	207	207
q15	521	457	452	452
q16	639	613	577	577
q17	595	887	347	347
q18	6732	6287	6208	6208
q19	1475	951	549	549
q20	332	325	188	188
q21	2960	2200	1986	1986
q22	369	332	303	303
Total cold run time: 102849 ms
Total hot run time: 31836 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5281	5154	5115	5115
q2	236	342	227	227
q3	2150	2708	2335	2335
q4	1452	1855	1403	1403
q5	4232	4155	4181	4155
q6	211	163	125	125
q7	1887	1808	1724	1724
q8	2641	2688	2522	2522
q9	7282	7312	7258	7258
q10	3010	3211	2820	2820
q11	582	529	494	494
q12	702	791	638	638
q13	3378	3852	3315	3315
q14	280	287	279	279
q15	501	468	508	468
q16	666	687	666	666
q17	1162	1672	1289	1289
q18	7653	7368	7327	7327
q19	869	927	1105	927
q20	1955	2039	1878	1878
q21	5579	5158	4794	4794
q22	652	594	530	530
Total cold run time: 52361 ms
Total hot run time: 50289 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191361 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bf11101155b91ddddb9b562016773c6ac0e2c7af, data reload: false

query1	1307	961	914	914
query2	6217	1866	1832	1832
query3	11179	4582	4643	4582
query4	56802	24897	23409	23409
query5	5257	515	488	488
query6	349	217	177	177
query7	5006	506	316	316
query8	323	240	225	225
query9	6212	2536	2550	2536
query10	394	302	246	246
query11	15266	14980	14922	14922
query12	153	106	103	103
query13	1133	527	368	368
query14	10194	6798	7228	6798
query15	195	199	181	181
query16	7097	666	488	488
query17	1082	703	561	561
query18	1542	410	317	317
query19	195	204	168	168
query20	130	121	143	121
query21	208	127	105	105
query22	4361	4323	4296	4296
query23	34129	33446	33536	33446
query24	5657	2437	2429	2429
query25	490	490	423	423
query26	668	289	165	165
query27	1686	506	339	339
query28	2962	2496	2456	2456
query29	624	559	419	419
query30	214	190	155	155
query31	933	902	770	770
query32	80	64	66	64
query33	494	353	307	307
query34	784	859	509	509
query35	798	848	723	723
query36	972	977	916	916
query37	127	104	78	78
query38	4208	4163	4282	4163
query39	1490	1430	1451	1430
query40	206	117	105	105
query41	53	53	52	52
query42	126	105	110	105
query43	509	513	475	475
query44	1359	821	809	809
query45	181	176	205	176
query46	888	1086	670	670
query47	1849	1864	1775	1775
query48	387	431	314	314
query49	688	520	431	431
query50	732	757	419	419
query51	4297	4364	4238	4238
query52	114	107	99	99
query53	234	269	194	194
query54	496	512	427	427
query55	82	80	78	78
query56	319	296	277	277
query57	1130	1186	1106	1106
query58	256	241	234	234
query59	2651	2941	2741	2741
query60	286	278	290	278
query61	125	124	139	124
query62	801	730	719	719
query63	234	196	193	193
query64	1493	1046	723	723
query65	3274	3244	3271	3244
query66	766	407	304	304
query67	15907	15794	15241	15241
query68	5335	914	520	520
query69	516	310	281	281
query70	1184	1158	1054	1054
query71	439	294	267	267
query72	6020	3688	3754	3688
query73	1229	759	356	356
query74	9044	8943	9033	8943
query75	3494	3152	2667	2667
query76	3644	1197	744	744
query77	563	371	278	278
query78	10150	10266	9412	9412
query79	1795	849	595	595
query80	680	538	451	451
query81	516	293	240	240
query82	387	122	98	98
query83	171	165	154	154
query84	288	90	70	70
query85	743	360	313	313
query86	339	303	300	300
query87	4344	4462	4429	4429
query88	2861	2265	2206	2206
query89	398	316	281	281
query90	1783	196	193	193
query91	137	142	112	112
query92	65	72	56	56
query93	2399	1051	584	584
query94	628	420	301	301
query95	350	270	258	258
query96	485	550	273	273
query97	3277	3404	3279	3279
query98	225	209	206	206
query99	1342	1383	1256	1256
Total cold run time: 295192 ms
Total hot run time: 191361 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.79 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bf11101155b91ddddb9b562016773c6ac0e2c7af, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.61	0.10	0.10
query5	0.55	0.61	0.54
query6	1.19	0.71	0.72
query7	0.02	0.01	0.02
query8	0.05	0.03	0.04
query9	0.60	0.53	0.52
query10	0.58	0.56	0.58
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.62	0.61	0.61
query14	2.80	2.83	2.70
query15	0.92	0.84	0.85
query16	0.38	0.38	0.37
query17	1.02	1.00	1.02
query18	0.20	0.20	0.19
query19	1.89	1.83	1.90
query20	0.02	0.01	0.01
query21	15.37	0.90	0.53
query22	0.75	1.24	0.73
query23	14.79	1.38	0.60
query24	7.18	0.86	0.80
query25	0.50	0.09	0.30
query26	0.54	0.16	0.13
query27	0.05	0.05	0.05
query28	9.13	0.90	0.44
query29	12.58	3.97	3.25
query30	0.26	0.09	0.06
query31	2.82	0.61	0.38
query32	3.23	0.54	0.46
query33	3.00	3.12	3.01
query34	15.78	5.14	4.55
query35	4.55	4.57	4.53
query36	0.66	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 104.79 s
Total hot run time: 30.79 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 0.00% (0/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.84% (12241/26704)
Line Coverage 35.34% (103482/292809)
Region Coverage 34.51% (53027/153653)
Branch Coverage 30.22% (26866/88914)

yiguolei pushed a commit that referenced this pull request Mar 5, 2025
#48606)

### What problem does this PR solve?
Related PR: #45041 

Problem Summary:
If we set hudi_jni_scanner to an incorrect value, jni_connector will be
null, causing a core dump.
So we set Spark Hudi JNI scanner as default and the hadoop hudi jni
reader will be supported in future
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 5, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Mar 5, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 5, 2025

PR approved by anyone and no changes requested.

@morningman morningman merged commit 528388a into apache:master Mar 5, 2025
30 of 31 checks passed
github-actions bot pushed a commit that referenced this pull request Mar 5, 2025
### What problem does this PR solve?
Related PR: #44267 

Problem Summary:
If we set hudi_jni_scanner to an incorrect value, jni_connector will be
null, causing a core dump.
So we set hadoop as default hudi jni scanner.
@suxiaogang223 suxiaogang223 deleted the fix_hudi_jni branch March 7, 2025 02:21
dataroaring pushed a commit that referenced this pull request Mar 20, 2025
…48712)

Cherry-picked from #48602

Co-authored-by: Socrates <suyiteng@selectdb.com>
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
### What problem does this PR solve?
Related PR: apache#44267 

Problem Summary:
If we set hudi_jni_scanner to an incorrect value, jni_connector will be
null, causing a core dump.
So we set hadoop as default hudi jni scanner.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.9-merged dev/3.0.5-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants