Skip to content

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Feb 25, 2025

What problem does this PR solve?

Related to apache/doris-thirdparty#267
Fix potential coredump like:

1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007F40E3728520 in /lib/x86_64-linux-gnu/libc.so.6
 4# getLastTLSExceptionRootCause at /home/zcp/repo_center/doris_branch-3.0/doris/thirdparty/src/doris-thirdparty-hadoop-3.3.6.3-for-doris/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:866
 5# doris::io::hdfs_error[abi:cxx11]() in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
 6# doris::io::HdfsFileWriter::append_hdfs_file(std::basic_string_view<char, std::char_traits<char> >) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:381
 7# doris::io::HdfsFileWriter::_flush_buffer() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:390
 8# doris::io::HdfsFileWriter::_close_impl() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:255
 9# doris::io::HdfsFileWriter::close(bool) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:248
10# doris::vectorized::ParquetOutputStream::Close() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/runtime/vparquet_transformer.cpp:84
11# doris::vectorized::ParquetOutputStream::~ParquetOutputStream() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/runtime/vparquet_transformer.cpp:57

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 25, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman
Copy link
Contributor Author

run buidall

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31580 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 06fee97321b996d5818bf1cd2acbeb753f88c8bd, data reload: false

------ Round 1 ----------------------------------
q1	17589	5186	5043	5043
q2	2044	302	167	167
q3	10412	1252	742	742
q4	10207	1020	527	527
q5	7532	2436	2306	2306
q6	192	164	132	132
q7	1009	748	617	617
q8	9310	1304	1100	1100
q9	4940	4949	4817	4817
q10	6882	2308	1879	1879
q11	481	277	263	263
q12	344	347	220	220
q13	17777	3862	3048	3048
q14	222	231	198	198
q15	529	460	460	460
q16	620	616	585	585
q17	583	854	329	329
q18	6562	6395	6144	6144
q19	1560	955	548	548
q20	316	325	184	184
q21	2814	2212	1976	1976
q22	370	329	295	295
Total cold run time: 102295 ms
Total hot run time: 31580 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5198	5109	5096	5096
q2	237	328	231	231
q3	2168	2733	2307	2307
q4	1426	1852	1363	1363
q5	4263	4102	4120	4102
q6	207	165	122	122
q7	1874	1868	1708	1708
q8	2605	2565	2596	2565
q9	7292	7212	7231	7212
q10	2996	3203	2786	2786
q11	583	527	493	493
q12	687	777	620	620
q13	3453	4015	3282	3282
q14	273	301	273	273
q15	509	471	464	464
q16	662	704	657	657
q17	1100	1505	1385	1385
q18	7633	7365	7326	7326
q19	770	827	927	827
q20	1947	2025	1893	1893
q21	5411	5140	4739	4739
q22	620	610	551	551
Total cold run time: 51914 ms
Total hot run time: 50002 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191240 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 06fee97321b996d5818bf1cd2acbeb753f88c8bd, data reload: false

query1	1289	953	918	918
query2	6214	1865	1862	1862
query3	11156	4607	4623	4607
query4	25728	23285	23329	23285
query5	3899	648	486	486
query6	287	206	205	205
query7	4013	521	304	304
query8	307	243	236	236
query9	8544	2578	2586	2578
query10	487	314	247	247
query11	15383	15078	14908	14908
query12	147	108	105	105
query13	1553	536	381	381
query14	9096	6942	6583	6583
query15	218	193	180	180
query16	7400	637	452	452
query17	1164	738	586	586
query18	1610	390	313	313
query19	195	202	181	181
query20	132	124	139	124
query21	245	124	105	105
query22	4516	4617	4508	4508
query23	34313	33495	33452	33452
query24	8157	2430	2430	2430
query25	514	472	405	405
query26	1214	277	153	153
query27	2215	525	342	342
query28	4294	2466	2496	2466
query29	631	563	457	457
query30	228	187	162	162
query31	944	901	807	807
query32	69	66	66	66
query33	541	380	307	307
query34	832	869	534	534
query35	809	872	753	753
query36	974	1011	899	899
query37	118	100	73	73
query38	4303	4138	4208	4138
query39	1476	1449	1490	1449
query40	216	126	111	111
query41	63	60	56	56
query42	123	112	114	112
query43	506	510	484	484
query44	1357	864	838	838
query45	183	177	166	166
query46	892	1073	670	670
query47	1855	1848	1775	1775
query48	402	449	326	326
query49	801	524	473	473
query50	695	761	412	412
query51	4277	4260	4277	4260
query52	114	116	104	104
query53	241	268	200	200
query54	510	503	416	416
query55	82	90	90	90
query56	280	274	251	251
query57	1201	1169	1127	1127
query58	248	250	238	238
query59	2919	2889	2886	2886
query60	310	270	263	263
query61	138	122	118	118
query62	783	744	663	663
query63	243	196	191	191
query64	4079	1073	780	780
query65	3315	3231	3213	3213
query66	1010	426	304	304
query67	16388	15565	15307	15307
query68	8908	927	516	516
query69	475	300	261	261
query70	1216	1143	1099	1099
query71	449	301	265	265
query72	5104	3604	3754	3604
query73	788	746	358	358
query74	8935	9015	8717	8717
query75	3900	3169	2723	2723
query76	3740	1174	735	735
query77	825	363	277	277
query78	10020	10409	9293	9293
query79	2776	833	597	597
query80	723	522	452	452
query81	504	278	245	245
query82	709	133	94	94
query83	182	168	149	149
query84	239	97	76	76
query85	854	353	310	310
query86	385	324	256	256
query87	4407	4500	4315	4315
query88	3530	2204	2196	2196
query89	420	326	287	287
query90	1816	196	197	196
query91	201	137	107	107
query92	77	61	55	55
query93	2066	1058	581	581
query94	636	427	303	303
query95	354	267	257	257
query96	481	559	264	264
query97	3384	3421	3304	3304
query98	221	206	201	201
query99	1583	1423	1266	1266
Total cold run time: 278354 ms
Total hot run time: 191240 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.85 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 06fee97321b996d5818bf1cd2acbeb753f88c8bd, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.07
query4	1.61	0.10	0.10
query5	0.58	0.54	0.55
query6	1.19	0.72	0.72
query7	0.03	0.02	0.02
query8	0.05	0.03	0.03
query9	0.58	0.54	0.53
query10	0.57	0.60	0.57
query11	0.16	0.11	0.10
query12	0.14	0.11	0.13
query13	0.61	0.60	0.60
query14	2.68	2.82	2.67
query15	0.92	0.86	0.86
query16	0.39	0.39	0.39
query17	1.02	1.06	1.00
query18	0.22	0.20	0.20
query19	1.87	1.78	1.97
query20	0.01	0.02	0.01
query21	15.35	0.90	0.55
query22	0.77	1.15	0.75
query23	14.86	1.42	0.61
query24	7.61	1.54	0.67
query25	0.50	0.25	0.12
query26	0.64	0.16	0.15
query27	0.05	0.06	0.05
query28	9.34	0.88	0.45
query29	12.56	4.01	3.31
query30	0.26	0.09	0.07
query31	2.80	0.61	0.39
query32	3.22	0.55	0.46
query33	3.11	2.98	3.04
query34	15.76	5.15	4.58
query35	4.55	4.57	4.60
query36	0.66	0.50	0.49
query37	0.09	0.06	0.07
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.18	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 105.53 s
Total hot run time: 30.85 s

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 25, 2025
@morningman morningman merged commit 248703b into apache:master Feb 25, 2025
33 checks passed
morningman added a commit to morningman/doris that referenced this pull request Feb 26, 2025
Related to apache/doris-thirdparty#267
Fix potential coredump like:

```
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007F40E3728520 in /lib/x86_64-linux-gnu/libc.so.6
 4# getLastTLSExceptionRootCause at /home/zcp/repo_center/doris_branch-3.0/doris/thirdparty/src/doris-thirdparty-hadoop-3.3.6.3-for-doris/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:866
 5# doris::io::hdfs_error[abi:cxx11]() in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
 6# doris::io::HdfsFileWriter::append_hdfs_file(std::basic_string_view<char, std::char_traits<char> >) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:381
 7# doris::io::HdfsFileWriter::_flush_buffer() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:390
 8# doris::io::HdfsFileWriter::_close_impl() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:255
 9# doris::io::HdfsFileWriter::close(bool) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:248
10# doris::vectorized::ParquetOutputStream::Close() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/runtime/vparquet_transformer.cpp:84
11# doris::vectorized::ParquetOutputStream::~ParquetOutputStream() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/runtime/vparquet_transformer.cpp:57
```
morningman added a commit to morningman/doris that referenced this pull request Feb 26, 2025
Related to apache/doris-thirdparty#267
Fix potential coredump like:

```
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007F40E3728520 in /lib/x86_64-linux-gnu/libc.so.6
 4# getLastTLSExceptionRootCause at /home/zcp/repo_center/doris_branch-3.0/doris/thirdparty/src/doris-thirdparty-hadoop-3.3.6.3-for-doris/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:866
 5# doris::io::hdfs_error[abi:cxx11]() in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
 6# doris::io::HdfsFileWriter::append_hdfs_file(std::basic_string_view<char, std::char_traits<char> >) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:381
 7# doris::io::HdfsFileWriter::_flush_buffer() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:390
 8# doris::io::HdfsFileWriter::_close_impl() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:255
 9# doris::io::HdfsFileWriter::close(bool) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:248
10# doris::vectorized::ParquetOutputStream::Close() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/runtime/vparquet_transformer.cpp:84
11# doris::vectorized::ParquetOutputStream::~ParquetOutputStream() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/runtime/vparquet_transformer.cpp:57
```
zhiqiang-hhhh pushed a commit to zhiqiang-hhhh/doris that referenced this pull request Feb 27, 2025
### What problem does this PR solve?

Related to apache/doris-thirdparty#267
Fix potential coredump like:

```
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007F40E3728520 in /lib/x86_64-linux-gnu/libc.so.6
 4# getLastTLSExceptionRootCause at /home/zcp/repo_center/doris_branch-3.0/doris/thirdparty/src/doris-thirdparty-hadoop-3.3.6.3-for-doris/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:866
 5# doris::io::hdfs_error[abi:cxx11]() in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
 6# doris::io::HdfsFileWriter::append_hdfs_file(std::basic_string_view<char, std::char_traits<char> >) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:381
 7# doris::io::HdfsFileWriter::_flush_buffer() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:390
 8# doris::io::HdfsFileWriter::_close_impl() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:255
 9# doris::io::HdfsFileWriter::close(bool) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:248
10# doris::vectorized::ParquetOutputStream::Close() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/runtime/vparquet_transformer.cpp:84
11# doris::vectorized::ParquetOutputStream::~ParquetOutputStream() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/runtime/vparquet_transformer.cpp:57
```
seawinde pushed a commit to seawinde/doris that referenced this pull request Feb 28, 2025
### What problem does this PR solve?

Related to apache/doris-thirdparty#267
Fix potential coredump like:

```
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007F40E3728520 in /lib/x86_64-linux-gnu/libc.so.6
 4# getLastTLSExceptionRootCause at /home/zcp/repo_center/doris_branch-3.0/doris/thirdparty/src/doris-thirdparty-hadoop-3.3.6.3-for-doris/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:866
 5# doris::io::hdfs_error[abi:cxx11]() in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
 6# doris::io::HdfsFileWriter::append_hdfs_file(std::basic_string_view<char, std::char_traits<char> >) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:381
 7# doris::io::HdfsFileWriter::_flush_buffer() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:390
 8# doris::io::HdfsFileWriter::_close_impl() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:255
 9# doris::io::HdfsFileWriter::close(bool) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:248
10# doris::vectorized::ParquetOutputStream::Close() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/runtime/vparquet_transformer.cpp:84
11# doris::vectorized::ParquetOutputStream::~ParquetOutputStream() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/runtime/vparquet_transformer.cpp:57
```
@gavinchou gavinchou mentioned this pull request Apr 23, 2025
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
### What problem does this PR solve?

Related to apache/doris-thirdparty#267
Fix potential coredump like:

```
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007F40E3728520 in /lib/x86_64-linux-gnu/libc.so.6
 4# getLastTLSExceptionRootCause at /home/zcp/repo_center/doris_branch-3.0/doris/thirdparty/src/doris-thirdparty-hadoop-3.3.6.3-for-doris/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:866
 5# doris::io::hdfs_error[abi:cxx11]() in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
 6# doris::io::HdfsFileWriter::append_hdfs_file(std::basic_string_view<char, std::char_traits<char> >) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:381
 7# doris::io::HdfsFileWriter::_flush_buffer() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:390
 8# doris::io::HdfsFileWriter::_close_impl() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:255
 9# doris::io::HdfsFileWriter::close(bool) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/io/fs/hdfs_file_writer.cpp:248
10# doris::vectorized::ParquetOutputStream::Close() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/runtime/vparquet_transformer.cpp:84
11# doris::vectorized::ParquetOutputStream::~ParquetOutputStream() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/runtime/vparquet_transformer.cpp:57
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.9-merged dev/3.0.5-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants