[Fix](load) fix load channel leak when load exception occurs #30915

liaoxin01 · 2024-02-06T09:13:53Z

Proposed changes

Issue Number: close #xxx

This was introduced by: #27111
When an exception occurs during load, only _cancel is set to true without calling the tablet_writer_cancel request, resulting in load channel not being released.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-02-06T09:13:57Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

liaoxin01 · 2024-02-06T09:17:37Z

run buildall

kaijchen

LGTM

github-actions · 2024-02-06T09:20:54Z

PR approved by anyone and no changes requested.

github-actions · 2024-02-06T09:23:20Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-02-06T09:36:30Z

TPC-H: Total hot run time: 37093 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5990153b972cf651b7fb4edcfc4c2eae7a049ed0, data reload: false

------ Round 1 ----------------------------------
q1	17603	4489	4512	4489
q2	2046	146	135	135
q3	10616	962	954	954
q4	4699	767	736	736
q5	7680	2748	2805	2748
q6	185	119	122	119
q7	1144	760	718	718
q8	9285	2000	2009	2000
q9	7303	6354	6371	6354
q10	8059	2442	2457	2442
q11	417	213	219	213
q12	764	280	276	276
q13	18033	3254	3298	3254
q14	275	260	246	246
q15	527	490	485	485
q16	468	416	415	415
q17	923	549	515	515
q18	6843	5976	5911	5911
q19	1547	1314	1426	1314
q20	631	343	320	320
q21	6572	3148	3193	3148
q22	795	312	301	301
Total cold run time: 106415 ms
Total hot run time: 37093 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4434	4403	4415	4403
q2	330	233	229	229
q3	2956	2833	2750	2750
q4	1846	1674	1703	1674
q5	5252	5261	5216	5216
q6	191	116	115	115
q7	2139	1752	1796	1752
q8	3123	3229	3237	3229
q9	8301	8302	8308	8302
q10	5768	3505	3525	3505
q11	536	457	461	457
q12	745	575	573	573
q13	13405	3111	3092	3092
q14	273	246	279	246
q15	531	482	487	482
q16	505	461	474	461
q17	1823	1671	1675	1671
q18	7840	7703	7573	7573
q19	6620	1508	1538	1508
q20	2122	1934	1934	1934
q21	4695	4537	4571	4537
q22	535	457	481	457
Total cold run time: 73970 ms
Total hot run time: 54166 ms

doris-robot · 2024-02-06T09:40:32Z

TeamCity be ut coverage result:
Function Coverage: 36.20% (8630/23839)
Line Coverage: 28.22% (70585/250126)
Region Coverage: 27.23% (36427/133787)
Branch Coverage: 24.00% (18654/77718)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5990153b972cf651b7fb4edcfc4c2eae7a049ed0_5990153b972cf651b7fb4edcfc4c2eae7a049ed0/report/index.html

doris-robot · 2024-02-06T09:47:45Z

TPC-DS: Total hot run time: 174495 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5990153b972cf651b7fb4edcfc4c2eae7a049ed0, data reload: false

query1	937	339	340	339
query2	6537	2017	1950	1950
query3	6695	215	207	207
query4	31786	22063	21985	21985
query5	4220	421	354	354
query6	252	163	170	163
query7	4599	289	288	288
query8	249	175	194	175
query9	8817	2277	2263	2263
query10	412	199	215	199
query11	18254	15535	15311	15311
query12	133	78	75	75
query13	1625	412	414	412
query14	9668	6799	6961	6799
query15	256	179	183	179
query16	8097	254	248	248
query17	1852	531	523	523
query18	2105	265	266	265
query19	364	141	142	141
query20	83	77	79	77
query21	195	131	121	121
query22	4879	4679	4721	4679
query23	31765	30086	30116	30086
query24	10470	2742	2746	2742
query25	578	356	351	351
query26	1480	144	148	144
query27	2922	299	302	299
query28	7585	1820	1826	1820
query29	927	603	611	603
query30	278	137	140	137
query31	903	717	720	717
query32	90	58	50	50
query33	742	216	217	216
query34	1102	460	473	460
query35	858	764	739	739
query36	1057	911	945	911
query37	115	61	61	61
query38	3208	3059	3100	3059
query39	1281	1253	1249	1249
query40	260	99	95	95
query41	37	35	37	35
query42	98	93	94	93
query43	505	472	476	472
query44	1081	685	704	685
query45	195	185	173	173
query46	1040	684	677	677
query47	1594	1532	1472	1472
query48	443	362	371	362
query49	1192	297	277	277
query50	755	377	380	377
query51	5235	5103	5177	5103
query52	106	89	90	89
query53	330	261	270	261
query54	279	219	220	219
query55	80	75	82	75
query56	225	204	205	204
query57	1008	851	947	851
query58	204	180	185	180
query59	2453	2452	2424	2424
query60	244	213	211	211
query61	92	90	82	82
query62	625	378	348	348
query63	297	257	269	257
query64	6253	3465	3776	3465
query65	3255	3254	3224	3224
query66	1054	308	302	302
query67	14494	14324	14177	14177
query68	4140	547	527	527
query69	455	334	335	334
query70	1268	1221	1250	1221
query71	322	269	247	247
query72	5954	2814	2658	2658
query73	695	320	325	320
query74	6537	6252	6219	6219
query75	3013	2393	2327	2327
query76	2518	971	947	947
query77	404	225	235	225
query78	9376	8656	8563	8563
query79	2622	500	503	500
query80	1801	357	348	348
query81	514	203	199	199
query82	676	83	81	81
query83	252	126	125	125
query84	277	82	75	75
query85	2175	331	333	331
query86	483	310	311	310
query87	3401	3197	3154	3154
query88	3873	2334	2318	2318
query89	443	354	348	348
query90	2071	166	162	162
query91	149	121	123	121
query92	55	43	45	43
query93	2298	503	454	454
query94	1281	179	177	177
query95	484	373	358	358
query96	603	272	260	260
query97	4216	4128	4103	4103
query98	221	194	189	189
query99	1089	664	694	664
Total cold run time: 284476 ms
Total hot run time: 174495 ms

doris-robot · 2024-02-06T09:53:11Z

ClickBench: Total hot run time: 31.39 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5990153b972cf651b7fb4edcfc4c2eae7a049ed0, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.02	0.02
query3	0.22	0.06	0.06
query4	1.67	0.10	0.10
query5	0.54	0.52	0.52
query6	1.18	0.63	0.63
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.54	0.51	0.49
query10	0.54	0.57	0.55
query11	0.11	0.08	0.08
query12	0.11	0.09	0.09
query13	0.61	0.62	0.60
query14	0.80	0.79	0.80
query15	0.80	0.78	0.79
query16	0.39	0.39	0.38
query17	1.03	1.05	1.00
query18	0.20	0.28	0.24
query19	1.88	1.76	1.83
query20	0.02	0.01	0.01
query21	15.40	0.56	0.56
query22	2.56	2.22	2.49
query23	17.13	0.83	0.76
query24	2.48	1.72	0.91
query25	0.31	0.13	0.19
query26	0.57	0.14	0.14
query27	0.05	0.04	0.04
query28	11.20	0.89	0.85
query29	12.60	3.26	3.25
query30	0.61	0.53	0.49
query31	2.78	0.35	0.36
query32	3.36	0.50	0.47
query33	3.21	3.24	3.21
query34	15.85	4.27	4.28
query35	4.30	4.28	4.31
query36	1.11	1.04	1.05
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.02	0.01
query40	0.16	0.13	0.13
query41	0.08	0.01	0.01
query42	0.02	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 104.73 s
Total hot run time: 31.39 s

doris-robot · 2024-02-06T09:56:16Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 5990153b972cf651b7fb4edcfc4c2eae7a049ed0 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.4 seconds inserted 10000000 Rows, about 694K ops/s

dataroaring

LGTM

github-actions · 2024-02-06T10:02:52Z

PR approved by at least one committer and no changes requested.

…30915)

…#30920)

…30915)

…30915) (apache#30920)

[Fix](load) fix load channel leak when load exception occurs

5990153

kaijchen approved these changes Feb 6, 2024

View reviewed changes

github-actions bot added the reviewed label Feb 6, 2024

xiaokang added p0_b dev/2.0.5 labels Feb 6, 2024

dataroaring approved these changes Feb 6, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 6, 2024

liaoxin01 added a commit to liaoxin01/doris that referenced this pull request Feb 6, 2024

[Fix](load) fix load channel leak when load exception occurs (apache#…

536a9ee

…30915)

liaoxin01 mentioned this pull request Feb 6, 2024

[Fix](load) fix load channel leak when load exception occurs (#30915) #30920

Merged

xiaokang merged commit a09486c into apache:master Feb 6, 2024

xiaokang pushed a commit that referenced this pull request Feb 6, 2024

[Fix](load) fix load channel leak when load exception occurs (#30915) (…

5a623d8

…#30920)

xiaokang added dev/2.0.5-merged and removed dev/2.0.5 labels Feb 6, 2024

yiguolei pushed a commit that referenced this pull request Feb 6, 2024

[Fix](load) fix load channel leak when load exception occurs (#30915)

84e7ed6

liaoxin01 deleted the fix_load_leak branch February 6, 2024 12:21

yiguolei pushed a commit that referenced this pull request Feb 16, 2024

[Fix](load) fix load channel leak when load exception occurs (#30915)

517e7a0

mymeiyi pushed a commit to mymeiyi/doris that referenced this pull request Feb 19, 2024

[Fix](load) fix load channel leak when load exception occurs (apache#…

c5dc272

…30915)

kaijchen mentioned this pull request Mar 29, 2024

[fix](load) skip cancel rpc if stub is not initialized #33006

Closed

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024

[Fix](load) fix load channel leak when load exception occurs (apache#…

5401bdb

…30915) (apache#30920)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fix](load) fix load channel leak when load exception occurs #30915

[Fix](load) fix load channel leak when load exception occurs #30915

Uh oh!

liaoxin01 commented Feb 6, 2024 •

edited

Loading

Uh oh!

doris-robot commented Feb 6, 2024

Uh oh!

liaoxin01 commented Feb 6, 2024

Uh oh!

kaijchen left a comment

Uh oh!

github-actions bot commented Feb 6, 2024

Uh oh!

github-actions bot commented Feb 6, 2024

Uh oh!

doris-robot commented Feb 6, 2024

Uh oh!

doris-robot commented Feb 6, 2024

Uh oh!

doris-robot commented Feb 6, 2024

Uh oh!

doris-robot commented Feb 6, 2024

Uh oh!

doris-robot commented Feb 6, 2024

Uh oh!

dataroaring left a comment

Uh oh!

github-actions bot commented Feb 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[Fix](load) fix load channel leak when load exception occurs #30915

[Fix](load) fix load channel leak when load exception occurs #30915

Uh oh!

Conversation

liaoxin01 commented Feb 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented Feb 6, 2024

Uh oh!

liaoxin01 commented Feb 6, 2024

Uh oh!

kaijchen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 6, 2024

Uh oh!

github-actions bot commented Feb 6, 2024

Uh oh!

doris-robot commented Feb 6, 2024

Uh oh!

doris-robot commented Feb 6, 2024

Uh oh!

doris-robot commented Feb 6, 2024

Uh oh!

doris-robot commented Feb 6, 2024

Uh oh!

doris-robot commented Feb 6, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

liaoxin01 commented Feb 6, 2024 •

edited

Loading