Skip to content

branch-3.0: [Fix](compaction) Fix nullptr in CloudStorageEngine due to concurrent access to compaction maps #50819 #50881

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
May 14, 2025
Merged

branch-3.0: [Fix](compaction) Fix nullptr in CloudStorageEngine due to concurrent access to compaction maps #50819 #50881

dataroaring merged 1 commit into from
May 14, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented May 14, 2025

Cherry-picked from #50819

@Yukang-Lian
[Fix](compaction) Fix nullptr in CloudStorageEngine due to concurrent…
c22cbed 
… access to compaction maps (#50819)

Related PR: #49882 

Problem Summary:

*** Query id: 0-0 ***
*** is nereids: 0 ***
*** tablet id: 0 ***
*** Aborted at 1746727905 (unix time) try "date -d @1746727905" if you
are using GNU date ***
*** Current BE git commitID: ace825a ***
*** SIGSEGV address not mapped to object (@0x8) received by PID 3151893
(TID 3152363 OR 0x7f1186c00640) from PID 8; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int,
siginfo_t*, void*) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0]
in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
2# JVM_handle_linux_signal in
/usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007F12D9FEE520 in /lib/x86_64-linux-gnu/libc.so.6
4# std::_Hashtable<long, std::pair<long const,
std::shared_ptr<doris::CloudBaseCompaction> >,
std::allocator<std::pair<long const,
std::shared_ptr<doris::CloudBaseCompaction> > >,
std::__detail::_Select1st, std::equal_to<long>, std::hash<long>,
std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash,
std::__detail::_Prime_rehash_policy,
std::__detail::_Hashtable_traits<false, false, true>
>::_M_find_before_node(unsigned long, long const&, unsigned long) const
at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/hashtable.h:1817
5# std::pair<std::__detail::_Node_iterator<std::pair<long const,
std::shared_ptr<doris::CloudBaseCompaction> >, false, false>, bool>
std::_Hashtable<long, std::pair<long const,
std::shared_ptr<doris::CloudBaseCompaction> >,
std::allocator<std::pair<long const,
std::shared_ptr<doris::CloudBaseCompaction> > >,
std::__detail::_Select1st, std::equal_to<long>, std::hash<long>,
std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash,
std::__detail::_Prime_rehash_policy,
std::__detail::_Hashtable_traits<false, false, true> >::_M_emplace<long,
decltype(nullptr)>(std::integral_constant<bool, true>, long&&,
decltype(nullptr)&&) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/hashtable.h:1947
6#
doris::CloudStorageEngine::_submit_base_compaction_task(std::shared_ptr<doris::CloudTablet>
const&) in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be
7#
doris::CloudStorageEngine::submit_compaction_task(std::shared_ptr<doris::CloudTablet>
const&, doris::CompactionType) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/cloud/cloud_storage_engine.cpp:917
8# doris::CloudStorageEngine::_compaction_tasks_producer_callback() at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/cloud/cloud_storage_engine.cpp:494
9# doris::Thread::supervise_thread(void*) at
/home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/thread.cpp:499
10# start_thread at ./nptl/pthread_create.c:442
11# 0x00007F12DA0D2850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
@github-actions github-actions bot requested a review from dataroaring as a code owner May 14, 2025 03:28
@hello-stephen
Copy link
Contributor

hello-stephen commented May 14, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this May 14, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented May 14, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 14, 2025

TPC-H: Total hot run time: 40110 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c22cbed304ee24008c18bd1011725f51defd69f8, data reload: false

------ Round 1 ----------------------------------
q1	17941	6856	6653	6653
q2	2955	168	162	162
q3	10558	1159	1145	1145
q4	10220	781	724	724
q5	7726	2883	2862	2862
q6	215	129	131	129
q7	985	617	602	602
q8	9364	1944	2003	1944
q9	6573	6386	6383	6383
q10	6966	2273	2266	2266
q11	457	273	263	263
q12	400	217	212	212
q13	17767	2982	2998	2982
q14	240	207	206	206
q15	506	468	472	468
q16	658	591	601	591
q17	976	551	648	551
q18	7191	6640	6739	6640
q19	1394	1093	1123	1093
q20	474	195	203	195
q21	4010	3315	3102	3102
q22	1104	937	967	937
Total cold run time: 108680 ms
Total hot run time: 40110 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6549	6624	6490	6490
q2	330	234	240	234
q3	2873	2757	3001	2757
q4	2116	1865	1728	1728
q5	5653	5696	5796	5696
q6	217	133	130	130
q7	2161	1784	1805	1784
q8	3348	3517	3527	3517
q9	8834	8786	8799	8786
q10	3578	3538	3489	3489
q11	596	509	495	495
q12	803	607	596	596
q13	7190	3124	3127	3124
q14	294	285	264	264
q15	523	459	463	459
q16	695	646	649	646
q17	1823	1602	1583	1583
q18	8186	7637	7738	7637
q19	1668	1534	1458	1458
q20	2132	1864	1884	1864
q21	5504	5304	5298	5298
q22	1072	998	947	947
Total cold run time: 66145 ms
Total hot run time: 58982 ms

@doris-robot
Copy link

doris-robot commented May 14, 2025

TPC-DS: Total hot run time: 190655 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c22cbed304ee24008c18bd1011725f51defd69f8, data reload: false

query1	971	378	371	371
query2	6520	2127	2052	2052
query3	6716	225	223	223
query4	34042	23407	23478	23407
query5	4336	457	460	457
query6	277	181	190	181
query7	4626	324	313	313
query8	270	213	205	205
query9	9560	2604	2577	2577
query10	485	263	264	263
query11	18082	15065	15138	15065
query12	160	106	102	102
query13	1640	432	422	422
query14	9333	7477	6552	6552
query15	259	175	179	175
query16	8123	467	501	467
query17	1620	563	546	546
query18	2138	308	311	308
query19	292	151	151	151
query20	119	106	109	106
query21	208	104	101	101
query22	4374	4111	4410	4111
query23	34519	33543	33697	33543
query24	11310	2878	2896	2878
query25	698	397	393	393
query26	1428	173	168	168
query27	2967	353	347	347
query28	8135	2395	2398	2395
query29	885	440	411	411
query30	327	163	163	163
query31	1004	777	817	777
query32	101	55	57	55
query33	781	290	271	271
query34	970	506	512	506
query35	866	693	751	693
query36	1070	936	931	931
query37	134	67	69	67
query38	3936	3846	3872	3846
query39	1495	1417	1446	1417
query40	287	107	107	107
query41	54	52	54	52
query42	121	107	102	102
query43	531	492	491	491
query44	1269	793	807	793
query45	185	176	176	176
query46	1125	721	718	718
query47	1945	1842	1872	1842
query48	471	400	393	393
query49	1211	422	411	411
query50	822	415	422	415
query51	7331	7187	7183	7183
query52	108	93	94	93
query53	266	188	194	188
query54	1254	471	478	471
query55	81	78	79	78
query56	274	263	245	245
query57	1248	1118	1099	1099
query58	248	215	228	215
query59	3172	3038	2983	2983
query60	294	265	275	265
query61	140	155	143	143
query62	874	673	695	673
query63	234	193	199	193
query64	5519	784	753	753
query65	3286	3203	3179	3179
query66	1465	327	351	327
query67	15873	15778	15670	15670
query68	4487	568	587	568
query69	444	295	276	276
query70	1141	1118	1109	1109
query71	337	263	269	263
query72	6641	4035	3993	3993
query73	754	342	347	342
query74	10367	9287	8835	8835
query75	3341	2657	2656	2656
query76	2733	1055	1097	1055
query77	408	288	273	273
query78	10610	9685	9541	9541
query79	2354	610	604	604
query80	1121	438	425	425
query81	562	240	241	240
query82	874	94	87	87
query83	232	147	142	142
query84	239	81	79	79
query85	1471	316	304	304
query86	455	280	298	280
query87	4343	4239	4263	4239
query88	4409	2393	2346	2346
query89	409	302	295	295
query90	1987	187	188	187
query91	185	152	150	150
query92	62	56	57	56
query93	1525	570	560	560
query94	918	301	290	290
query95	360	258	258	258
query96	624	281	283	281
query97	3347	3146	3140	3140
query98	213	211	193	193
query99	1480	1351	1314	1314
Total cold run time: 302898 ms
Total hot run time: 190655 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented May 14, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/2) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 40.97% (10809/26381)
Line Coverage 31.81% (92292/290095)
Region Coverage 30.90% (47648/154194)
Branch Coverage 27.40% (24417/89100)

@doris-robot
Copy link

doris-robot commented May 14, 2025

ClickBench: Total hot run time: 31.25 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c22cbed304ee24008c18bd1011725f51defd69f8, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.22	0.07	0.06
query4	1.63	0.11	0.11
query5	0.54	0.50	0.50
query6	1.14	0.72	0.72
query7	0.02	0.01	0.02
query8	0.03	0.03	0.03
query9	0.57	0.50	0.51
query10	0.56	0.55	0.59
query11	0.14	0.10	0.13
query12	0.13	0.12	0.11
query13	0.60	0.60	0.59
query14	2.83	2.74	2.89
query15	0.90	0.83	0.83
query16	0.38	0.40	0.38
query17	1.06	0.98	1.00
query18	0.25	0.21	0.21
query19	1.87	1.84	2.10
query20	0.01	0.01	0.01
query21	15.37	0.59	0.58
query22	2.92	1.96	1.51
query23	17.20	0.92	0.85
query24	2.94	0.45	1.83
query25	0.29	0.11	0.06
query26	0.44	0.14	0.14
query27	0.05	0.04	0.04
query28	10.40	0.50	0.51
query29	12.61	3.23	3.22
query30	0.25	0.06	0.06
query31	2.86	0.40	0.37
query32	3.24	0.48	0.45
query33	3.01	3.02	3.01
query34	16.53	4.47	4.49
query35	4.51	4.44	4.56
query36	0.66	0.48	0.47
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.74 s
Total hot run time: 31.25 s

dataroaring
dataroaring approved these changes May 14, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit ff8b554 into branch-3.0 May 14, 2025
22 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-50819-branch-3.0 branch May 14, 2025 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@hello-stephen @doris-robot @dataroaring @Yukang-Lian