Skip to content

[fix](build) Skip some hdfsBuilder functions when USE_HADOOP_HDFS support is not enabled #49529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
zy-kkk merged 1 commit into from
Mar 31, 2025
Merged

[fix](build) Skip some hdfsBuilder functions when USE_HADOOP_HDFS support is not enabled #49529

zy-kkk merged 1 commit into from
Mar 31, 2025

Conversation

@zy-kkk
Copy link
Member

@zy-kkk zy-kkk commented Mar 26, 2025

We should check whether the USE_HADOOP_HDFS macro is enabled before deciding whether to use the hdfsBuilderSetKerb5Conf and hdfsBuilderSetKeyTabFile functions. Otherwise, compilation may fail in some environments that do not have HADOOP_HDFS compilation dependencies, such as MacOS.

@Thearas
Copy link
Contributor

Thearas commented Mar 26, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zy-kkk
Copy link
Member Author

zy-kkk commented Mar 26, 2025

run buildall

morningman
morningman approved these changes Mar 26, 2025
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 26, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Mar 26, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 26, 2025

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Mar 26, 2025

TPC-H: Total hot run time: 34240 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2bb982755b9fbf3a4d1a0e47f5d68386326b6cb7, data reload: false

------ Round 1 ----------------------------------
q1	26184	5394	5127	5127
q2	2086	290	162	162
q3	10412	1221	671	671
q4	10219	1006	547	547
q5	7562	2355	2308	2308
q6	190	162	134	134
q7	906	763	604	604
q8	9313	1301	1082	1082
q9	6722	5086	5134	5086
q10	6808	2289	1899	1899
q11	486	283	255	255
q12	357	364	221	221
q13	17778	3707	3128	3128
q14	241	229	216	216
q15	529	492	484	484
q16	641	620	587	587
q17	588	858	350	350
q18	7580	7253	7211	7211
q19	1523	959	549	549
q20	320	351	192	192
q21	3967	2609	2461	2461
q22	1061	1015	966	966
Total cold run time: 115473 ms
Total hot run time: 34240 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5314	5170	5178	5170
q2	243	328	228	228
q3	2113	2624	2272	2272
q4	1453	1822	1419	1419
q5	4537	4504	4359	4359
q6	215	165	127	127
q7	1960	1888	1698	1698
q8	2561	2628	2478	2478
q9	7343	7131	7046	7046
q10	2965	3184	2751	2751
q11	573	511	494	494
q12	665	732	596	596
q13	3632	4031	3298	3298
q14	286	279	268	268
q15	509	474	489	474
q16	657	660	644	644
q17	1126	1521	1406	1406
q18	7602	7537	7534	7534
q19	820	831	926	831
q20	1973	2029	1864	1864
q21	5299	4843	4586	4586
q22	1037	993	945	945
Total cold run time: 52883 ms
Total hot run time: 50488 ms

@doris-robot
Copy link

doris-robot commented Mar 26, 2025

TPC-DS: Total hot run time: 186371 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2bb982755b9fbf3a4d1a0e47f5d68386326b6cb7, data reload: false

query1	1040	495	482	482
query2	6546	1938	1877	1877
query3	6789	219	220	219
query4	26408	23663	23418	23418
query5	4320	661	461	461
query6	292	196	190	190
query7	4609	491	265	265
query8	289	249	225	225
query9	8620	2587	2595	2587
query10	475	312	273	273
query11	15422	15116	14866	14866
query12	148	111	104	104
query13	1642	520	397	397
query14	8758	6162	6201	6162
query15	194	190	170	170
query16	7143	644	432	432
query17	939	703	566	566
query18	1967	394	302	302
query19	189	179	157	157
query20	121	122	114	114
query21	206	122	108	108
query22	4082	4154	3950	3950
query23	33836	33121	33020	33020
query24	8403	2366	2379	2366
query25	543	444	434	434
query26	1231	267	141	141
query27	2771	490	316	316
query28	4415	2406	2374	2374
query29	764	549	427	427
query30	288	218	190	190
query31	940	862	784	784
query32	77	65	65	65
query33	560	360	318	318
query34	784	838	498	498
query35	780	816	749	749
query36	969	988	908	908
query37	122	106	79	79
query38	4083	4262	4152	4152
query39	1467	1406	1391	1391
query40	217	123	108	108
query41	61	61	59	59
query42	124	105	107	105
query43	501	510	484	484
query44	1320	801	814	801
query45	183	177	168	168
query46	849	1026	630	630
query47	1789	1829	1713	1713
query48	403	432	301	301
query49	799	541	449	449
query50	682	722	431	431
query51	4179	4216	4127	4127
query52	110	109	100	100
query53	229	254	187	187
query54	510	502	415	415
query55	88	84	93	84
query56	319	295	277	277
query57	1154	1130	1090	1090
query58	260	254	249	249
query59	2649	2744	2560	2560
query60	296	274	270	270
query61	137	130	129	129
query62	822	719	653	653
query63	222	182	180	180
query64	4438	1005	666	666
query65	4407	4332	4287	4287
query66	1147	409	307	307
query67	15797	15681	15287	15287
query68	8949	880	509	509
query69	483	310	264	264
query70	1215	1119	1126	1119
query71	463	298	260	260
query72	5734	4807	4862	4807
query73	724	657	355	355
query74	9001	9123	8814	8814
query75	4211	3229	2723	2723
query76	3615	1194	730	730
query77	791	369	288	288
query78	9967	10074	9284	9284
query79	6274	802	546	546
query80	712	535	448	448
query81	475	261	215	215
query82	724	130	97	97
query83	220	184	161	161
query84	289	105	79	79
query85	856	356	313	313
query86	337	322	302	302
query87	4434	4605	4450	4450
query88	2917	2246	2273	2246
query89	454	322	276	276
query90	1917	216	217	216
query91	140	154	113	113
query92	78	62	56	56
query93	3562	1058	567	567
query94	674	447	329	329
query95	361	275	263	263
query96	498	586	280	280
query97	3201	3195	3108	3108
query98	228	208	200	200
query99	1443	1387	1299	1299
Total cold run time: 280871 ms
Total hot run time: 186371 ms

@doris-robot
Copy link

doris-robot commented Mar 26, 2025

ClickBench: Total hot run time: 31.31 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2bb982755b9fbf3a4d1a0e47f5d68386326b6cb7, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.11	0.11
query3	0.25	0.19	0.19
query4	1.59	0.19	0.12
query5	0.55	0.54	0.55
query6	1.21	0.72	0.71
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.57	0.52	0.50
query10	0.61	0.58	0.58
query11	0.16	0.11	0.12
query12	0.14	0.11	0.11
query13	0.63	0.61	0.61
query14	2.67	2.72	2.68
query15	0.95	0.84	0.85
query16	0.40	0.39	0.39
query17	1.03	1.04	0.99
query18	0.21	0.20	0.20
query19	2.07	1.89	1.90
query20	0.01	0.01	0.01
query21	15.45	0.88	0.54
query22	0.75	1.22	0.90
query23	14.70	1.37	0.64
query24	7.03	1.46	0.94
query25	0.54	0.16	0.06
query26	0.61	0.17	0.13
query27	0.05	0.05	0.05
query28	9.27	0.88	0.43
query29	12.52	3.97	3.28
query30	0.25	0.08	0.06
query31	2.84	0.58	0.39
query32	3.23	0.55	0.47
query33	3.07	3.09	3.01
query34	15.73	5.12	4.46
query35	4.55	4.56	4.51
query36	0.66	0.48	0.48
query37	0.09	0.06	0.07
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.14	0.12
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.03 s
Total hot run time: 31.31 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 26, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/3) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 50.89% (13619/26763)
Line Coverage 40.27% (118210/293571)
Region Coverage 38.95% (60062/154209)
Branch Coverage 33.86% (30214/89224)

@zy-kkk zy-kkk merged commit f3493e0 into apache:master Mar 31, 2025
30 of 33 checks passed
@zy-kkk zy-kkk deleted the fix_build_0326_1 branch March 31, 2025 02:43
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@zy-kkk
[fix](build) Skip some hdfsBuilder functions when USE_HADOOP_HDFS sup…
6dc00e2 
…port is not enabled (apache#49529)

We should check whether the USE_HADOOP_HDFS macro is enabled before
deciding whether to use the hdfsBuilderSetKerb5Conf and
hdfsBuilderSetKeyTabFile functions. Otherwise, compilation may fail in
some environments that do not have HADOOP_HDFS compilation dependencies,
such as MacOS.
@morningman morningman mentioned this pull request Jun 29, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@gavinchou gavinchou gavinchou approved these changes

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@dataroaring dataroaring Awaiting requested review from dataroaring dataroaring is a code owner

+1 more reviewer

@swjtu-zhanglei swjtu-zhanglei swjtu-zhanglei approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.0-merged p0_test reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@zy-kkk @Thearas @doris-robot @hello-stephen @morningman @gavinchou @CalvinKirs @swjtu-zhanglei