Skip to content

[fix](deps) upgrade hadoop to 3.3.6.5 (#48280) #48391

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Feb 28, 2025
Merged

[fix](deps) upgrade hadoop to 3.3.6.5 (#48280) #48391

dataroaring merged 1 commit into from
Feb 28, 2025

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Feb 26, 2025

bp #48280

@morningman
[fix](deps) upgrade hadoop to 3.3.6.5 (apache#48280)
d835eb7 
Related to apache/doris-thirdparty#267
Fix potential coredump like:

```
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007F40E3728520 in /lib/x86_64-linux-gnu/libc.so.6
 4# getLastTLSExceptionRootCause at /home/zcp/repo_center/doris_branch-3.0/doris/thirdparty/src/doris-thirdparty-hadoop-3.3.6.3-for-doris/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:866
 5# doris::io::hdfs_error[abi:cxx11]() in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
 6# doris::io::HdfsFileWriter::append_hdfs_file(std::basic_string_view<char, std::char_traits<char> >) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:381
 7# doris::io::HdfsFileWriter::_flush_buffer() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:390
 8# doris::io::HdfsFileWriter::_close_impl() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:255
 9# doris::io::HdfsFileWriter::close(bool) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:248
10# doris::vectorized::ParquetOutputStream::Close() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/runtime/vparquet_transformer.cpp:84
11# doris::vectorized::ParquetOutputStream::~ParquetOutputStream() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/runtime/vparquet_transformer.cpp:57
```
@hello-stephen
Copy link
Contributor

hello-stephen commented Feb 26, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman
Copy link
Contributor Author

morningman commented Feb 26, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Feb 26, 2025

TPC-H: Total hot run time: 40093 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d835eb7e902d049f79fa2cd36e76434c8a100538, data reload: false

------ Round 1 ----------------------------------
q1	17586	6682	6620	6620
q2	2043	172	154	154
q3	10767	1070	1162	1070
q4	10529	771	761	761
q5	7748	2875	2777	2777
q6	218	134	129	129
q7	935	620	610	610
q8	9369	1930	2041	1930
q9	6597	6395	6381	6381
q10	7293	2314	2314	2314
q11	482	279	276	276
q12	427	214	217	214
q13	18371	3091	3029	3029
q14	256	213	214	213
q15	510	471	469	469
q16	655	577	593	577
q17	988	592	584	584
q18	7429	6753	6686	6686
q19	1395	1089	994	994
q20	489	203	199	199
q21	3984	3128	3277	3128
q22	1110	997	978	978
Total cold run time: 109181 ms
Total hot run time: 40093 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6584	6582	6610	6582
q2	329	227	228	227
q3	2888	2758	2946	2758
q4	2037	1832	1840	1832
q5	5751	5708	5688	5688
q6	210	127	126	126
q7	2238	1811	1808	1808
q8	3377	3626	3472	3472
q9	8858	8886	8781	8781
q10	3590	3557	3528	3528
q11	582	486	500	486
q12	820	629	579	579
q13	9148	3224	3153	3153
q14	307	270	281	270
q15	527	467	477	467
q16	687	640	652	640
q17	1831	1602	1591	1591
q18	8340	7715	7733	7715
q19	1632	1485	1573	1485
q20	2076	1871	1887	1871
q21	5535	5392	5430	5392
q22	1134	1064	1031	1031
Total cold run time: 68481 ms
Total hot run time: 59482 ms

@doris-robot
Copy link

doris-robot commented Feb 26, 2025

TPC-DS: Total hot run time: 197635 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d835eb7e902d049f79fa2cd36e76434c8a100538, data reload: false

query1	1291	906	897	897
query2	6244	2039	1963	1963
query3	10908	4408	4396	4396
query4	61477	28951	23154	23154
query5	5273	463	440	440
query6	418	182	177	177
query7	5404	313	309	309
query8	304	231	230	230
query9	8296	2655	2623	2623
query10	475	265	254	254
query11	17677	15450	15649	15450
query12	153	101	102	101
query13	1404	456	420	420
query14	10459	7430	7185	7185
query15	197	187	176	176
query16	7087	512	467	467
query17	1194	557	575	557
query18	1967	323	316	316
query19	208	157	161	157
query20	116	112	114	112
query21	205	102	104	102
query22	4985	4833	4771	4771
query23	34657	34035	34047	34035
query24	6148	2895	2881	2881
query25	531	410	424	410
query26	659	171	169	169
query27	1801	351	353	351
query28	4362	2489	2476	2476
query29	738	488	461	461
query30	244	174	168	168
query31	1031	827	832	827
query32	66	55	57	55
query33	473	310	294	294
query34	915	509	523	509
query35	865	742	735	735
query36	1113	974	950	950
query37	116	78	70	70
query38	4158	4054	3959	3959
query39	1544	1527	1477	1477
query40	199	96	95	95
query41	49	47	49	47
query42	117	100	102	100
query43	535	483	493	483
query44	1164	825	803	803
query45	184	167	174	167
query46	1164	739	728	728
query47	2068	1979	1978	1978
query48	462	398	390	390
query49	721	426	396	396
query50	849	436	416	416
query51	7399	7294	7213	7213
query52	105	92	92	92
query53	265	191	194	191
query54	565	471	462	462
query55	85	79	72	72
query56	268	236	284	236
query57	1304	1206	1216	1206
query58	230	217	224	217
query59	3224	2948	3028	2948
query60	277	252	241	241
query61	109	119	108	108
query62	764	683	658	658
query63	210	184	185	184
query64	1382	676	624	624
query65	3224	3205	3170	3170
query66	717	296	295	295
query67	16185	15580	15687	15580
query68	4129	577	561	561
query69	424	264	263	263
query70	1174	1017	1149	1017
query71	358	254	257	254
query72	6299	4050	4097	4050
query73	739	355	343	343
query74	10379	9015	9145	9015
query75	3334	2596	2627	2596
query76	2009	1044	1044	1044
query77	502	290	289	289
query78	10627	9686	9632	9632
query79	1440	603	587	587
query80	887	426	422	422
query81	517	242	235	235
query82	1281	89	86	86
query83	243	147	136	136
query84	282	77	77	77
query85	878	300	288	288
query86	322	295	292	292
query87	4451	4252	4360	4252
query88	3755	2390	2373	2373
query89	417	296	293	293
query90	2016	208	183	183
query91	182	149	147	147
query92	60	49	48	48
query93	1960	556	562	556
query94	742	295	262	262
query95	348	247	245	245
query96	613	281	282	281
query97	3312	3156	3170	3156
query98	219	201	198	198
query99	1580	1292	1270	1270
Total cold run time: 314467 ms
Total hot run time: 197635 ms

@doris-robot
Copy link

doris-robot commented Feb 26, 2025

ClickBench: Total hot run time: 32.22 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d835eb7e902d049f79fa2cd36e76434c8a100538, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.63	0.11	0.10
query5	0.54	0.51	0.52
query6	1.13	0.71	0.72
query7	0.02	0.01	0.01
query8	0.05	0.03	0.03
query9	0.56	0.49	0.49
query10	0.55	0.55	0.54
query11	0.14	0.10	0.11
query12	0.14	0.14	0.11
query13	0.60	0.61	0.60
query14	2.85	2.78	2.77
query15	0.88	0.82	0.82
query16	0.38	0.39	0.38
query17	1.05	1.05	1.08
query18	0.24	0.22	0.21
query19	1.85	1.91	2.03
query20	0.02	0.01	0.01
query21	15.36	0.59	0.57
query22	2.51	2.38	1.74
query23	17.00	0.96	0.91
query24	3.14	0.80	0.66
query25	0.15	0.27	0.08
query26	0.38	0.13	0.14
query27	0.05	0.05	0.04
query28	10.96	0.50	0.47
query29	12.62	3.30	3.28
query30	0.25	0.06	0.07
query31	2.85	0.38	0.37
query32	3.28	0.45	0.46
query33	2.98	3.01	3.05
query34	16.76	4.62	4.58
query35	4.59	4.60	4.66
query36	0.65	0.47	0.53
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.13
query41	0.07	0.02	0.02
query42	0.03	0.03	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.96 s
Total hot run time: 32.22 s

dataroaring
dataroaring approved these changes Feb 28, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 4472d72 into apache:branch-3.0 Feb 28, 2025
23 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@morningman @hello-stephen @doris-robot @dataroaring