Skip to content

[fix](cloud) fix filecache warmup crash due to spurious wakeup #48623

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
gavinchou merged 1 commit into from
Mar 4, 2025
Merged

[fix](cloud) fix filecache warmup crash due to spurious wakeup #48623

gavinchou merged 1 commit into from
Mar 4, 2025

Conversation

@freemandealer
Copy link
Contributor

@freemandealer freemandealer commented Mar 4, 2025 •
edited
Loading

What problem does this PR solve?

*** SIGSEGV address not mapped to object (@0x30) received by PID 1379 (TID 1888 OR 0x7fa73d4c8700) from PID 48; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /opt/jdk-17/lib/server/libjvm.so
2# JVM_handle_linux_signal in /opt/jdk-17/lib/server/libjvm.so
3# 0x00007FAB4C4ED400 in /lib64/libc.so.6
4# doris::CloudWarmUpManager::handle_jobs() at /home/zcp/repo_center/doris_release/doris/be/src/cloud/cloud_warm_up_manager.cpp:73

_pending_job_metas may wake up with empty, i.e. spurious wakeup

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@freemandealer
[fix](cloud) fix filecache warmup crash due to spurious wakeup
97d04fa 
Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 4, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@freemandealer
Copy link
Contributor Author

freemandealer commented Mar 4, 2025

run buildall

@dataroaring dataroaring added dev/3.0.x usercase Important user case type label labels Mar 4, 2025
@doris-robot
Copy link

doris-robot commented Mar 4, 2025

TPC-H: Total hot run time: 31549 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 97d04fa82274c8fcb30d8f6d78d96300ade5f7c2, data reload: false

------ Round 1 ----------------------------------
q1	17572	5256	5084	5084
q2	2047	299	170	170
q3	10427	1230	764	764
q4	10304	1010	530	530
q5	8679	2359	2419	2359
q6	194	172	141	141
q7	917	760	605	605
q8	9317	1330	1072	1072
q9	5011	4632	4958	4632
q10	6868	2298	1867	1867
q11	484	272	258	258
q12	344	360	235	235
q13	17763	3683	3065	3065
q14	232	226	209	209
q15	514	463	434	434
q16	633	620	584	584
q17	596	848	345	345
q18	6597	6165	6220	6165
q19	2139	960	546	546
q20	319	308	191	191
q21	2796	2252	1994	1994
q22	364	337	299	299
Total cold run time: 104117 ms
Total hot run time: 31549 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5253	5175	5124	5124
q2	234	330	235	235
q3	2153	2675	2296	2296
q4	1377	1803	1374	1374
q5	4236	4148	4131	4131
q6	209	167	128	128
q7	1875	1861	1778	1778
q8	2646	2563	2603	2563
q9	7275	7263	7180	7180
q10	3013	3253	2751	2751
q11	570	515	513	513
q12	704	791	639	639
q13	3357	3915	3323	3323
q14	276	292	286	286
q15	514	489	465	465
q16	669	678	643	643
q17	1134	1604	1353	1353
q18	7461	7297	7312	7297
q19	796	804	939	804
q20	2003	2046	1863	1863
q21	5405	5005	4679	4679
q22	616	562	567	562
Total cold run time: 51776 ms
Total hot run time: 49987 ms

@doris-robot
Copy link

doris-robot commented Mar 4, 2025

TPC-DS: Total hot run time: 191142 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 97d04fa82274c8fcb30d8f6d78d96300ade5f7c2, data reload: false

query1	1314	934	966	934
query2	6152	1901	1884	1884
query3	11183	4536	4551	4536
query4	54556	24455	23297	23297
query5	5206	550	490	490
query6	341	187	171	171
query7	4883	510	309	309
query8	307	238	225	225
query9	5856	2555	2542	2542
query10	411	312	268	268
query11	15018	15042	15121	15042
query12	153	107	109	107
query13	1063	531	383	383
query14	10381	6577	6864	6577
query15	208	212	193	193
query16	7013	679	508	508
query17	1112	733	588	588
query18	1535	426	315	315
query19	199	212	174	174
query20	129	129	133	129
query21	210	129	117	117
query22	4673	4696	4479	4479
query23	33958	33332	33279	33279
query24	5740	2449	2385	2385
query25	474	471	418	418
query26	810	272	163	163
query27	1971	504	336	336
query28	2817	2496	2469	2469
query29	600	556	421	421
query30	211	191	154	154
query31	910	870	813	813
query32	70	63	60	60
query33	459	368	316	316
query34	797	879	510	510
query35	807	869	764	764
query36	948	1006	875	875
query37	127	105	74	74
query38	4189	4302	4236	4236
query39	1486	1415	1438	1415
query40	203	111	109	109
query41	51	48	48	48
query42	124	109	110	109
query43	496	524	483	483
query44	1300	844	825	825
query45	178	176	167	167
query46	900	1081	645	645
query47	1879	1904	1816	1816
query48	411	416	308	308
query49	728	534	410	410
query50	709	759	425	425
query51	4339	4270	4214	4214
query52	110	102	103	102
query53	238	262	184	184
query54	495	495	427	427
query55	84	83	84	83
query56	265	269	258	258
query57	1176	1209	1119	1119
query58	237	242	241	241
query59	2677	2806	2768	2768
query60	291	283	306	283
query61	125	118	144	118
query62	741	744	677	677
query63	231	198	185	185
query64	2183	1035	680	680
query65	3335	3258	3263	3258
query66	769	397	303	303
query67	15907	15708	15367	15367
query68	6881	889	506	506
query69	536	305	260	260
query70	1239	1142	1129	1129
query71	487	291	325	291
query72	5913	3550	3779	3550
query73	1376	750	356	356
query74	8914	9214	8723	8723
query75	3789	3135	2813	2813
query76	4266	1181	732	732
query77	610	357	300	300
query78	10183	10080	9294	9294
query79	2556	835	594	594
query80	704	523	449	449
query81	506	271	244	244
query82	559	123	93	93
query83	182	168	158	158
query84	284	88	73	73
query85	810	349	310	310
query86	383	287	301	287
query87	4414	4406	4343	4343
query88	3492	2223	2219	2219
query89	420	317	278	278
query90	1817	193	191	191
query91	141	149	110	110
query92	80	56	60	56
query93	2091	1050	578	578
query94	660	409	275	275
query95	346	267	257	257
query96	487	564	273	273
query97	3356	3407	3320	3320
query98	232	207	200	200
query99	1461	1403	1312	1312
Total cold run time: 297634 ms
Total hot run time: 191142 ms

@freemandealer
Copy link
Contributor Author

freemandealer commented Mar 4, 2025

run cloud_p0

@doris-robot
Copy link

doris-robot commented Mar 4, 2025

ClickBench: Total hot run time: 31.04 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 97d04fa82274c8fcb30d8f6d78d96300ade5f7c2, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.03
query3	0.24	0.07	0.06
query4	1.61	0.10	0.10
query5	0.57	0.57	0.56
query6	1.16	0.72	0.71
query7	0.02	0.02	0.01
query8	0.05	0.03	0.03
query9	0.60	0.55	0.52
query10	0.57	0.58	0.57
query11	0.16	0.11	0.11
query12	0.14	0.10	0.11
query13	0.62	0.59	0.60
query14	2.69	2.69	2.70
query15	0.92	0.85	0.84
query16	0.36	0.39	0.39
query17	1.06	1.06	1.04
query18	0.21	0.19	0.19
query19	1.92	1.81	1.97
query20	0.02	0.01	0.01
query21	15.36	0.92	0.57
query22	0.76	1.22	0.68
query23	14.86	1.39	0.62
query24	7.15	1.62	0.90
query25	0.49	0.16	0.13
query26	0.64	0.16	0.15
query27	0.06	0.05	0.05
query28	9.12	0.88	0.45
query29	12.52	3.97	3.30
query30	0.25	0.09	0.07
query31	2.82	0.61	0.38
query32	3.22	0.54	0.46
query33	2.97	3.03	3.04
query34	15.84	5.14	4.54
query35	4.57	4.53	4.54
query36	0.66	0.49	0.48
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.18	0.14	0.13
query41	0.09	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.02
Total cold run time: 104.83 s
Total hot run time: 31.04 s

@doris-robot
Copy link

doris-robot commented Mar 4, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/8) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.84% (12241/26706)
Line Coverage 35.34% (103499/292835)
Region Coverage 34.52% (53044/153682)
Branch Coverage 30.23% (26877/88920)

@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Mar 4, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2025

PR approved by anyone and no changes requested.

@gavinchou gavinchou added the p0_c label Mar 4, 2025
@gavinchou gavinchou merged commit 9b94385 into apache:master Mar 4, 2025
29 of 32 checks passed
github-actions bot pushed a commit that referenced this pull request Mar 4, 2025
@freemandealer
[fix](cloud) fix filecache warmup crash due to spurious wakeup (#48623)
374d09f 
*** SIGSEGV address not mapped to object (@0x30) received by PID 1379
(TID 1888 OR 0x7fa73d4c8700) from PID 48; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int,
siginfo_t*, void*) at
/home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0]
in /opt/jdk-17/lib/server/libjvm.so
 2# JVM_handle_linux_signal in /opt/jdk-17/lib/server/libjvm.so
 3# 0x00007FAB4C4ED400 in /lib64/libc.so.6
4# doris::CloudWarmUpManager::handle_jobs() at
/home/zcp/repo_center/doris_release/doris/be/src/cloud/cloud_warm_up_manager.cpp:73

_pending_job_metas may wake up with empty, i.e. spurious wakeup
@github-actions github-actions bot mentioned this pull request Mar 4, 2025
Merged
dataroaring pushed a commit that referenced this pull request Mar 10, 2025
@github-actions @freemandealer
branch-3.0: [fix](cloud) fix filecache warmup crash due to spurious w…
5dfcabf 
…akeup #48623 (#48654)

Cherry-picked from #48623

Co-authored-by: zhengyu <zhangzhengyu@selectdb.com>
@gavinchou gavinchou mentioned this pull request Apr 23, 2025
Closed
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@freemandealer
[fix](cloud) fix filecache warmup crash due to spurious wakeup (apach…
88fcd08 
…e#48623)

*** SIGSEGV address not mapped to object (@0x30) received by PID 1379
(TID 1888 OR 0x7fa73d4c8700) from PID 48; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int,
siginfo_t*, void*) at
/home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0]
in /opt/jdk-17/lib/server/libjvm.so
 2# JVM_handle_linux_signal in /opt/jdk-17/lib/server/libjvm.so
 3# 0x00007FAB4C4ED400 in /lib64/libc.so.6
4# doris::CloudWarmUpManager::handle_jobs() at
/home/zcp/repo_center/doris_release/doris/be/src/cloud/cloud_warm_up_manager.cpp:73

_pending_job_metas may wake up with empty, i.e. spurious wakeup
@XLPE XLPE mentioned this pull request Jun 27, 2025
Closed
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.5-merged p0_c reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@freemandealer @hello-stephen @doris-robot @gavinchou @dataroaring