Skip to content

[Pick 2.0] "Fix data loss when node channel been cancelled before close wait (#36662)" #36740

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jun 25, 2024
Merged

[Pick 2.0] "Fix data loss when node channel been cancelled before close wait (#36662)" #36740

dataroaring merged 1 commit into from
Jun 25, 2024

Conversation

@platoneko
Copy link
Contributor

@platoneko platoneko commented Jun 24, 2024

Proposed changes

Pick from #36662

@doris-robot
Copy link

doris-robot commented Jun 24, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@platoneko platoneko marked this pull request as ready for review June 24, 2024 07:08
@github-actions
Copy link
Contributor

github-actions bot commented Jun 24, 2024

clang-tidy review says "All clean, LGTM! 👍"

@platoneko platoneko changed the title Pick "Fix data loss when node channel been cancelled before close wait (#36662)" [Pick 2.1] "Fix data loss when node channel been cancelled before close wait (#36662)" Jun 24, 2024
@platoneko platoneko changed the title [Pick 2.1] "Fix data loss when node channel been cancelled before close wait (#36662)" [Pick 2.0] "Fix data loss when node channel been cancelled before close wait (#36662)" Jun 24, 2024
@dataroaring
Copy link
Contributor

dataroaring commented Jun 24, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jun 24, 2024

TeamCity be ut coverage result:
Function Coverage: 37.85% (8105/21415)
Line Coverage: 29.52% (66336/224726)
Region Coverage: 28.98% (34182/117936)
Branch Coverage: 24.85% (17544/70608)
Coverage Report: http://coverage.selectdb-in.cc/coverage/592a8e10f6291267f93a016261c48e913702bb0a_592a8e10f6291267f93a016261c48e913702bb0a/report/index.html

@doris-robot
Copy link

doris-robot commented Jun 24, 2024

TPC-H: Total hot run time: 49870 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 592a8e10f6291267f93a016261c48e913702bb0a, data reload: false

------ Round 1 ----------------------------------
q1	18163	4367	4329	4329
q2	2068	154	146	146
q3	10404	1891	1955	1891
q4	10392	1225	1307	1225
q5	8886	3856	3894	3856
q6	276	127	126	126
q7	2018	1586	1593	1586
q8	9276	2716	2683	2683
q9	10659	10451	10389	10389
q10	8623	3460	3519	3460
q11	426	233	247	233
q12	469	301	305	301
q13	19735	3959	4084	3959
q14	353	334	328	328
q15	524	470	460	460
q16	708	575	587	575
q17	1140	954	987	954
q18	7329	6839	6849	6839
q19	1777	1680	1667	1667
q20	521	324	324	324
q21	4475	4140	4090	4090
q22	547	449	454	449
Total cold run time: 118769 ms
Total hot run time: 49870 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4372	4357	4504	4357
q2	356	248	261	248
q3	4350	4295	4319	4295
q4	2812	2823	2787	2787
q5	7342	7378	7257	7257
q6	239	125	121	121
q7	3272	2849	2858	2849
q8	4431	4480	4595	4480
q9	16851	16673	16792	16673
q10	4229	4219	4224	4219
q11	750	689	682	682
q12	1034	852	878	852
q13	6914	3761	3728	3728
q14	476	415	432	415
q15	507	457	469	457
q16	740	681	670	670
q17	3901	3871	3813	3813
q18	8752	8681	8825	8681
q19	1700	1713	1667	1667
q20	2339	2176	2115	2115
q21	8411	8475	8430	8430
q22	1080	1003	982	982
Total cold run time: 84858 ms
Total hot run time: 79778 ms

@doris-robot
Copy link

doris-robot commented Jun 24, 2024

TPC-DS: Total hot run time: 203216 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 592a8e10f6291267f93a016261c48e913702bb0a, data reload: false

query1	951	423	380	380
query2	6528	3022	2716	2716
query3	6936	219	200	200
query4	19997	17940	17951	17940
query5	19733	6512	6491	6491
query6	291	225	223	223
query7	4161	309	311	309
query8	409	402	384	384
query9	3092	2659	2594	2594
query10	414	300	296	296
query11	11339	10668	10715	10668
query12	132	78	80	78
query13	5613	686	673	673
query14	17719	13419	13198	13198
query15	366	250	239	239
query16	6477	275	257	257
query17	1723	1469	861	861
query18	2322	412	414	412
query19	217	148	150	148
query20	78	80	80	80
query21	196	102	99	99
query22	5257	5051	5041	5041
query23	32597	32040	32072	32040
query24	6997	6506	6518	6506
query25	535	423	422	422
query26	518	164	165	164
query27	1776	306	300	300
query28	6158	2351	2303	2303
query29	2951	2771	2709	2709
query30	242	165	170	165
query31	895	758	766	758
query32	64	61	59	59
query33	402	286	260	260
query34	846	495	489	489
query35	1109	888	950	888
query36	1329	1135	1079	1079
query37	89	63	62	62
query38	3099	2919	2921	2919
query39	1389	1340	1316	1316
query40	207	104	95	95
query41	45	45	44	44
query42	81	86	87	86
query43	770	729	707	707
query44	1138	725	730	725
query45	246	236	235	235
query46	1234	980	977	977
query47	1877	1782	1659	1659
query48	1013	705	696	696
query49	622	379	371	371
query50	883	628	631	628
query51	4713	4633	4659	4633
query52	90	83	87	83
query53	445	334	319	319
query54	2656	2453	2483	2453
query55	84	80	88	80
query56	229	222	215	215
query57	1250	1116	1090	1090
query58	227	192	195	192
query59	4104	3933	3868	3868
query60	219	188	200	188
query61	96	91	101	91
query62	814	471	440	440
query63	489	340	348	340
query64	2523	1556	1313	1313
query65	3624	3542	3547	3542
query66	749	381	392	381
query67	17312	16241	15399	15399
query68	6428	663	654	654
query69	570	356	363	356
query70	1845	1296	1456	1296
query71	398	306	323	306
query72	6521	3439	3469	3439
query73	729	328	318	318
query74	6425	5924	5925	5924
query75	4522	3743	3795	3743
query76	4242	1127	1149	1127
query77	535	257	250	250
query78	12662	11937	11552	11552
query79	9116	654	645	645
query80	2923	403	400	400
query81	522	243	240	240
query82	1778	98	98	98
query83	193	132	133	132
query84	262	72	72	72
query85	1425	323	315	315
query86	329	284	296	284
query87	3221	3073	3011	3011
query88	5329	2287	2296	2287
query89	379	294	278	278
query90	1657	213	209	209
query91	173	141	140	140
query92	62	57	53	53
query93	5799	597	572	572
query94	639	217	218	217
query95	1099	1053	1035	1035
query96	642	325	321	321
query97	6442	6261	6421	6261
query98	195	173	182	173
query99	2962	986	932	932
Total cold run time: 312711 ms
Total hot run time: 203216 ms

@doris-robot
Copy link

doris-robot commented Jun 24, 2024

ClickBench: Total hot run time: 30.92 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 592a8e10f6291267f93a016261c48e913702bb0a, data reload: false

query1	0.02	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.05	0.04
query4	1.79	0.05	0.06
query5	0.53	0.52	0.52
query6	1.23	0.62	0.68
query7	0.01	0.01	0.00
query8	0.03	0.02	0.02
query9	0.52	0.48	0.48
query10	0.53	0.53	0.52
query11	0.12	0.08	0.09
query12	0.11	0.10	0.09
query13	0.61	0.61	0.62
query14	0.78	0.79	0.79
query15	0.80	0.75	0.76
query16	0.36	0.36	0.36
query17	1.01	1.01	1.03
query18	0.23	0.24	0.24
query19	1.96	1.76	1.86
query20	0.02	0.01	0.01
query21	15.51	0.55	0.53
query22	2.10	2.36	1.42
query23	17.35	0.96	1.02
query24	3.62	1.87	1.50
query25	0.32	0.13	0.05
query26	0.55	0.18	0.16
query27	0.05	0.04	0.04
query28	8.07	0.74	0.71
query29	12.65	2.34	2.23
query30	0.55	0.53	0.54
query31	2.81	0.38	0.37
query32	3.41	0.50	0.49
query33	3.09	3.11	3.08
query34	15.23	4.82	4.76
query35	4.84	4.83	4.83
query36	1.07	1.01	1.02
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.13	0.14
query41	0.06	0.01	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 102.56 s
Total hot run time: 30.92 s

@doris-robot
Copy link

doris-robot commented Jun 24, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 592a8e10f6291267f93a016261c48e913702bb0a with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

@github-actions
Copy link
Contributor

github-actions bot commented Jun 24, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jun 24, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jun 24, 2024

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit d163979 into apache:branch-2.0 Jun 25, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@platoneko
[Pick 2.0] "Fix data loss when node channel been cancelled before clo…
fa7c541 
…se wait (apache#36662)" (apache#36740)

## Proposed changes
Pick from apache#36662
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@platoneko @doris-robot @dataroaring @gavinchou