Skip to content

[fix](load) skip sending cancel rpc if VNodeChannel is not inited #34897

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
May 15, 2024
Merged

[fix](load) skip sending cancel rpc if VNodeChannel is not inited #34897

dataroaring merged 1 commit into from
May 15, 2024

Conversation

@kaijchen
Copy link
Member

@kaijchen kaijchen commented May 15, 2024

Proposed changes

Fix the following coredump

Core was generated by `/mnt/hdd01/DorisCloudS3Stress/cluster0/be/lib/doris_be'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  pthread_sigmask (how=2, newmask=<optimized out>, oldmask=0x0) at pthread_sigmask.c:48
48      pthread_sigmask.c: No such file or directory.
[Current thread is 1 (Thread 0x7fdbf91f1700 (LWP 3649251))]
(gdb) bt
#0  pthread_sigmask (how=2, newmask=<optimized out>, oldmask=0x0) at pthread_sigmask.c:48
#1  0x00007ffa17acd71e in PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] ()
   from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
#2  0x00007ffa17ace206 in JVM_handle_linux_signal () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
#3  <signal handler called>
#4  doris::vectorized::VNodeChannel::cancel (this=this@entry=0x7ff4ef107190, cancel_msg=...)
    at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/sink/writer/vtablet_writer.cpp:912
#5  0x00005646777a937b in doris::vectorized::VTabletWriter::_cancel_all_channel(doris::Status)::$_0::operator()(std::shared_ptr<doris::vectorized::VNodeChannel> const&) const (this=<optimized out>, ch=...)
    at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/sink/writer/vtablet_writer.cpp:1388
#6  std::__invoke_impl<void, doris::vectorized::VTabletWriter::_cancel_all_channel(doris::Status)::$_0&, std::shared_ptr<doris::vectorized::VNodeChannel> const&>(std::__invoke_other, doris::vectorized::VTabletWriter::_cancel_all_channel(doris::Status)::$_0&, std::shared_ptr<doris::vectorized::VNodeChannel> const&) (__f=..., __args=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61

...

(gdb) f 4
#4  doris::vectorized::VNodeChannel::cancel (this=this@entry=0x7ff4ef107190, cancel_msg=...)
    at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/sink/writer/vtablet_writer.cpp:912
912     /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/sink/writer/vtablet_writer.cpp: No such file or directory.
(gdb) p this
$1 = (doris::vectorized::VNodeChannel *) 0x7ff4ef107190
(gdb) p _stub
$2 = {<std::__shared_ptr<doris::PBackendService_Stub, (__gnu_cxx::_Lock_policy)2>> = {<std::__shared_ptr_access<doris::PBackendService_Stub, (__gnu_cxx::_Lock_policy)2, false, false>> = {<No data fields>}, _M_ptr = 0x0, _M_refcount = {_M_pi = 0x0}}, <No data fields>}
(gdb) p _inited
$3 = false

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented May 15, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

dataroaring
dataroaring approved these changes May 15, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kaijchen
Copy link
Member Author

kaijchen commented May 15, 2024

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 15, 2024
@github-actions
Copy link
Contributor

github-actions bot commented May 15, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented May 15, 2024

PR approved by anyone and no changes requested.

@kaijchen kaijchen mentioned this pull request May 15, 2024
Closed
@github-actions
Copy link
Contributor

github-actions bot commented May 15, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented May 15, 2024

TPC-H: Total hot run time: 40385 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ce4d549eca5bdcbeaaf4a5ef0235385243b5276c, data reload: false

------ Round 1 ----------------------------------
q1	17620	4257	4204	4204
q2	2022	205	197	197
q3	10438	1233	1198	1198
q4	10157	839	692	692
q5	7493	2722	2648	2648
q6	217	133	130	130
q7	962	548	520	520
q8	9233	2100	2089	2089
q9	9072	6512	6466	6466
q10	8924	3673	3734	3673
q11	457	240	235	235
q12	475	217	221	217
q13	17914	3044	3012	3012
q14	255	219	215	215
q15	527	473	473	473
q16	483	384	398	384
q17	961	647	717	647
q18	8167	7604	7418	7418
q19	1633	1566	1519	1519
q20	639	304	306	304
q21	5137	3949	3867	3867
q22	361	277	295	277
Total cold run time: 113147 ms
Total hot run time: 40385 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4264	4220	4206	4206
q2	373	280	281	280
q3	3012	2801	2756	2756
q4	1842	1651	1591	1591
q5	5303	5281	5284	5281
q6	209	123	120	120
q7	1708	1359	1375	1359
q8	3189	3342	3348	3342
q9	8389	8283	8304	8283
q10	3853	3645	3670	3645
q11	583	499	490	490
q12	766	575	617	575
q13	17551	2954	2970	2954
q14	303	259	268	259
q15	530	468	473	468
q16	472	419	415	415
q17	1783	1507	1493	1493
q18	7623	7547	7318	7318
q19	1649	1514	1508	1508
q20	1952	1757	1786	1757
q21	5093	4858	4843	4843
q22	544	494	480	480
Total cold run time: 70991 ms
Total hot run time: 53423 ms

@doris-robot
Copy link

doris-robot commented May 15, 2024

TeamCity be ut coverage result:
Function Coverage: 35.65% (8982/25194)
Line Coverage: 27.31% (74256/271854)
Region Coverage: 26.55% (38371/144541)
Branch Coverage: 23.37% (19575/83754)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ce4d549eca5bdcbeaaf4a5ef0235385243b5276c_ce4d549eca5bdcbeaaf4a5ef0235385243b5276c/report/index.html

@doris-robot
Copy link

doris-robot commented May 15, 2024

TPC-DS: Total hot run time: 188813 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ce4d549eca5bdcbeaaf4a5ef0235385243b5276c, data reload: false

query1	940	392	374	374
query2	6469	2365	2404	2365
query3	6660	209	209	209
query4	23081	21459	21176	21176
query5	4177	423	421	421
query6	268	195	172	172
query7	4589	302	295	295
query8	247	191	190	190
query9	8488	2410	2413	2410
query10	442	260	244	244
query11	14785	14242	14199	14199
query12	128	93	90	90
query13	1635	368	361	361
query14	10124	8495	8292	8292
query15	258	173	174	173
query16	8120	261	260	260
query17	1868	553	551	551
query18	2085	278	274	274
query19	212	147	145	145
query20	93	88	86	86
query21	203	131	138	131
query22	5049	4821	4838	4821
query23	34283	33389	33762	33389
query24	11145	2958	2946	2946
query25	599	382	373	373
query26	708	159	162	159
query27	2220	335	339	335
query28	6124	2088	2051	2051
query29	860	638	614	614
query30	314	174	174	174
query31	1002	790	743	743
query32	92	56	53	53
query33	756	246	240	240
query34	1046	480	490	480
query35	814	698	683	683
query36	1103	899	939	899
query37	111	70	71	70
query38	2979	2815	2807	2807
query39	1605	1558	1560	1558
query40	193	126	121	121
query41	49	43	42	42
query42	102	100	101	100
query43	583	550	554	550
query44	1253	716	743	716
query45	270	263	256	256
query46	1094	731	733	731
query47	1972	1890	1889	1889
query48	372	295	318	295
query49	1063	395	438	395
query50	767	379	396	379
query51	6899	6801	6859	6801
query52	102	91	95	91
query53	349	289	280	280
query54	861	435	430	430
query55	76	71	71	71
query56	239	225	235	225
query57	1270	1168	1145	1145
query58	223	204	207	204
query59	3312	3278	3387	3278
query60	260	232	240	232
query61	91	93	91	91
query62	655	466	474	466
query63	309	275	310	275
query64	8660	7518	7390	7390
query65	3111	3093	3088	3088
query66	810	345	344	344
query67	15428	15053	15032	15032
query68	4580	544	560	544
query69	478	320	307	307
query70	1135	1148	1167	1148
query71	430	276	291	276
query72	7948	2692	2470	2470
query73	714	334	335	334
query74	6532	6174	6123	6123
query75	3399	2659	2665	2659
query76	2859	1028	937	937
query77	470	293	274	274
query78	10827	10313	10164	10164
query79	2095	522	540	522
query80	1008	467	455	455
query81	548	245	245	245
query82	672	100	97	97
query83	255	175	191	175
query84	251	89	90	89
query85	1688	330	314	314
query86	495	291	306	291
query87	3317	3134	3148	3134
query88	4329	2421	2426	2421
query89	481	382	394	382
query90	1967	195	191	191
query91	137	109	112	109
query92	63	52	57	52
query93	2278	530	509	509
query94	1201	190	195	190
query95	414	309	313	309
query96	599	278	269	269
query97	3138	2995	3011	2995
query98	244	227	208	208
query99	1221	890	899	890
Total cold run time: 281828 ms
Total hot run time: 188813 ms

@doris-robot
Copy link

doris-robot commented May 15, 2024

ClickBench: Total hot run time: 30.16 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ce4d549eca5bdcbeaaf4a5ef0235385243b5276c, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.03
query3	0.23	0.05	0.05
query4	1.67	0.07	0.07
query5	0.49	0.47	0.50
query6	1.11	0.72	0.73
query7	0.01	0.01	0.01
query8	0.05	0.04	0.04
query9	0.53	0.47	0.49
query10	0.54	0.56	0.53
query11	0.15	0.11	0.11
query12	0.15	0.12	0.11
query13	0.59	0.59	0.60
query14	0.79	0.78	0.77
query15	0.83	0.81	0.82
query16	0.37	0.37	0.36
query17	1.02	1.01	1.02
query18	0.23	0.23	0.24
query19	1.77	1.80	1.81
query20	0.02	0.01	0.00
query21	15.46	0.70	0.65
query22	4.05	8.26	1.58
query23	18.27	1.34	1.23
query24	1.52	0.36	0.21
query25	0.13	0.08	0.08
query26	0.25	0.17	0.17
query27	0.08	0.09	0.08
query28	13.36	1.02	1.01
query29	13.12	3.26	3.30
query30	0.24	0.08	0.07
query31	2.83	0.38	0.40
query32	3.28	0.47	0.46
query33	2.78	2.81	2.84
query34	17.31	4.43	4.46
query35	4.47	4.48	4.49
query36	0.64	0.46	0.46
query37	0.18	0.15	0.15
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.16	0.14	0.14
query41	0.09	0.04	0.04
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.19 s
Total hot run time: 30.16 s

@dataroaring dataroaring merged commit 197ee08 into apache:master May 15, 2024
yiguolei pushed a commit that referenced this pull request May 18, 2024
@kaijchen @Doris-Extras
[fix](load) skip sending cancel rpc if VNodeChannel is not inited (#3…
849eeb3 
…4897)
kaijchen added a commit to kaijchen/doris that referenced this pull request Jul 10, 2024
@kaijchen
[fix](load) skip sending cancel rpc if VNodeChannel is not inited (ap…
e1e9644 
…ache#34897)
@kaijchen kaijchen mentioned this pull request Jul 10, 2024
Merged
dataroaring pushed a commit that referenced this pull request Jul 10, 2024
@kaijchen
[fix](load) skip sending cancel rpc if VNodeChannel is not inited (#3…
13e6e8a 
…4897) (#37577)

cherry-pick #34897
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@kaijchen
[fix](load) skip sending cancel rpc if VNodeChannel is not inited (ap…
3b15652 
…ache#34897) (apache#37577)

cherry-pick apache#34897
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.13-merged dev/2.1.3-merged dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kaijchen @doris-robot @gavinchou @dataroaring @wm1581066