[fix](move-memtable) fix load timeout caused by lost wakeup #32720

liaoxin01 · 2024-03-23T03:34:57Z

Proposed changes

Issue Number: close #xxx

introduced by：#31689

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-03-23T03:35:02Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

liaoxin01 · 2024-03-23T03:35:13Z

run buildall

github-actions · 2024-03-23T03:40:12Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-03-23T04:04:18Z

TeamCity be ut coverage result:
Function Coverage: 35.26% (8733/24766)
Line Coverage: 27.06% (71513/264294)
Region Coverage: 26.29% (37097/141096)
Branch Coverage: 23.19% (18975/81816)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4ce0cafedaaa260de5c6992cc58f760e8439064d_4ce0cafedaaa260de5c6992cc58f760e8439064d/report/index.html

doris-robot · 2024-03-23T04:13:00Z

TPC-H: Total hot run time: 37820 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4ce0cafedaaa260de5c6992cc58f760e8439064d, data reload: false

------ Round 1 ----------------------------------
q1	17677	4222	4093	4093
q2	2139	158	151	151
q3	10596	1180	1151	1151
q4	10224	786	792	786
q5	7460	3027	2997	2997
q6	200	124	121	121
q7	1023	583	571	571
q8	9340	1962	1967	1962
q9	7614	6595	6502	6502
q10	8477	3459	3523	3459
q11	430	220	217	217
q12	410	199	199	199
q13	17816	2884	2849	2849
q14	251	210	197	197
q15	517	474	462	462
q16	503	377	372	372
q17	935	625	525	525
q18	7128	6545	6361	6361
q19	4655	1390	1384	1384
q20	563	252	249	249
q21	3531	2922	3070	2922
q22	351	291	290	290
Total cold run time: 111840 ms
Total hot run time: 37820 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4148	4059	4076	4059
q2	328	227	225	225
q3	2994	2906	2834	2834
q4	1817	1571	1547	1547
q5	5289	5315	5312	5312
q6	194	113	115	113
q7	2222	1825	1828	1825
q8	3148	3284	3279	3279
q9	8696	8730	8686	8686
q10	3784	3794	3774	3774
q11	537	441	437	437
q12	731	571	568	568
q13	16926	2841	2874	2841
q14	280	252	258	252
q15	497	458	463	458
q16	467	418	408	408
q17	1732	1499	1472	1472
q18	7493	7179	7134	7134
q19	1609	1488	1474	1474
q20	1904	1715	1720	1715
q21	4860	4584	4604	4584
q22	522	461	439	439
Total cold run time: 70178 ms
Total hot run time: 53436 ms

doris-robot · 2024-03-23T04:24:59Z

TPC-DS: Total hot run time: 185620 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4ce0cafedaaa260de5c6992cc58f760e8439064d, data reload: false

query1	957	359	367	359
query2	7491	2102	1955	1955
query3	6702	214	209	209
query4	31810	21249	21516	21249
query5	4309	406	388	388
query6	285	187	183	183
query7	4624	286	285	285
query8	223	172	170	170
query9	9208	2254	2249	2249
query10	567	243	260	243
query11	17368	14453	14380	14380
query12	143	95	90	90
query13	1652	408	405	405
query14	12001	11180	10511	10511
query15	273	210	194	194
query16	8165	254	259	254
query17	1953	584	584	584
query18	2105	293	284	284
query19	272	155	149	149
query20	99	89	90	89
query21	209	128	125	125
query22	4939	4813	4782	4782
query23	33426	32696	32727	32696
query24	10713	2866	2891	2866
query25	623	393	384	384
query26	1221	159	155	155
query27	2562	343	349	343
query28	7367	1891	1847	1847
query29	891	657	638	638
query30	301	153	149	149
query31	950	740	713	713
query32	94	66	62	62
query33	774	257	334	257
query34	1026	491	493	491
query35	844	625	638	625
query36	1032	926	914	914
query37	126	83	76	76
query38	3613	3572	3495	3495
query39	1472	1445	1417	1417
query40	212	117	115	115
query41	50	48	46	46
query42	102	95	98	95
query43	465	452	448	448
query44	1114	697	720	697
query45	285	263	270	263
query46	1132	713	702	702
query47	1942	1830	1813	1813
query48	443	347	346	346
query49	1131	346	327	327
query50	755	362	363	362
query51	6741	6482	6796	6482
query52	105	91	91	91
query53	340	269	275	269
query54	316	246	235	235
query55	81	80	76	76
query56	237	225	269	225
query57	1205	1128	1134	1128
query58	223	197	201	197
query59	2832	2729	2742	2729
query60	255	246	252	246
query61	96	108	91	91
query62	680	449	445	445
query63	298	272	274	272
query64	5698	3945	3821	3821
query65	3056	3033	3016	3016
query66	867	364	366	364
query67	15019	14657	15006	14657
query68	6829	516	511	511
query69	605	372	377	372
query70	1227	1144	1171	1144
query71	499	283	269	269
query72	6603	2695	2527	2527
query73	724	309	303	303
query74	7112	6646	6586	6586
query75	4110	2985	3011	2985
query76	4470	854	793	793
query77	632	265	262	262
query78	10733	10160	10111	10111
query79	7996	512	506	506
query80	1501	403	402	402
query81	540	219	218	218
query82	911	214	203	203
query83	203	151	150	150
query84	298	87	82	82
query85	1556	368	359	359
query86	456	332	297	297
query87	3754	3533	3554	3533
query88	4880	2273	2274	2273
query89	523	360	360	360
query90	2001	169	173	169
query91	177	129	132	129
query92	63	47	46	46
query93	5981	494	480	480
query94	1138	174	172	172
query95	419	318	325	318
query96	602	269	266	266
query97	3041	2862	2851	2851
query98	230	216	206	206
query99	1180	935	914	914
Total cold run time: 309064 ms
Total hot run time: 185620 ms

doris-robot · 2024-03-23T04:28:13Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 4ce0cafedaaa260de5c6992cc58f760e8439064d with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

liaoxin01 · 2024-03-23T04:31:13Z

run buildall

github-actions · 2024-03-23T04:36:09Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-03-23T04:48:49Z

TPC-H: Total hot run time: 38627 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f14d53e24f2c3c21940587d5dd726642c7affffe, data reload: false

------ Round 1 ----------------------------------
q1	17778	4650	4204	4204
q2	3106	170	160	160
q3	11358	1169	1214	1169
q4	10876	788	834	788
q5	7531	3095	3051	3051
q6	211	128	126	126
q7	1061	634	617	617
q8	9581	2053	2032	2032
q9	7390	6774	6657	6657
q10	8441	3466	3588	3466
q11	441	233	220	220
q12	378	203	199	199
q13	17790	2877	2881	2877
q14	251	202	208	202
q15	508	449	466	449
q16	509	384	389	384
q17	977	594	575	575
q18	7363	6570	6548	6548
q19	1599	1456	1494	1456
q20	539	251	262	251
q21	3730	2956	2893	2893
q22	353	303	314	303
Total cold run time: 111771 ms
Total hot run time: 38627 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4116	4075	4176	4075
q2	335	236	237	236
q3	3023	2885	2839	2839
q4	1936	1605	1612	1605
q5	5465	5454	5398	5398
q6	201	117	119	117
q7	2297	1868	1912	1868
q8	3201	3368	3346	3346
q9	8907	8874	8877	8874
q10	3878	3808	3768	3768
q11	560	443	442	442
q12	709	550	543	543
q13	16895	2880	2855	2855
q14	287	247	262	247
q15	506	470	461	461
q16	484	443	432	432
q17	1737	1508	1485	1485
q18	7480	7502	7092	7092
q19	1598	1539	1506	1506
q20	1934	1736	1718	1718
q21	5019	4725	4847	4725
q22	540	474	466	466
Total cold run time: 71108 ms
Total hot run time: 54098 ms

doris-robot · 2024-03-23T04:59:36Z

TeamCity be ut coverage result:
Function Coverage: 35.26% (8733/24766)
Line Coverage: 27.06% (71505/264294)
Region Coverage: 26.29% (37100/141096)
Branch Coverage: 23.19% (18971/81816)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f14d53e24f2c3c21940587d5dd726642c7affffe_f14d53e24f2c3c21940587d5dd726642c7affffe/report/index.html

doris-robot · 2024-03-23T05:00:55Z

TPC-DS: Total hot run time: 187520 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f14d53e24f2c3c21940587d5dd726642c7affffe, data reload: false

query1	962	373	367	367
query2	7373	2044	1963	1963
query3	6719	214	219	214
query4	32009	21570	21397	21397
query5	4267	444	408	408
query6	275	177	178	177
query7	4625	301	293	293
query8	233	166	180	166
query9	9210	2280	2273	2273
query10	574	255	257	255
query11	15569	14544	14718	14544
query12	135	94	87	87
query13	1629	414	407	407
query14	14981	11447	11648	11447
query15	295	203	194	194
query16	8221	262	257	257
query17	2010	596	551	551
query18	2062	306	284	284
query19	351	163	165	163
query20	94	89	93	89
query21	218	131	130	130
query22	5026	4816	4766	4766
query23	33558	32868	32846	32846
query24	11017	2823	2843	2823
query25	598	379	367	367
query26	1351	156	155	155
query27	2966	368	364	364
query28	7241	1851	1831	1831
query29	877	637	602	602
query30	301	149	148	148
query31	956	725	742	725
query32	94	60	52	52
query33	767	249	239	239
query34	1016	474	496	474
query35	813	617	595	595
query36	1027	889	880	880
query37	116	73	78	73
query38	3555	3465	3422	3422
query39	1448	1456	1419	1419
query40	214	109	109	109
query41	47	43	43	43
query42	104	97	94	94
query43	486	454	459	454
query44	1160	734	725	725
query45	274	244	249	244
query46	1135	720	711	711
query47	1948	1830	1827	1827
query48	433	358	357	357
query49	1096	339	339	339
query50	772	370	371	370
query51	6769	6533	6613	6533
query52	115	88	96	88
query53	348	278	277	277
query54	298	242	246	242
query55	95	79	81	79
query56	232	234	232	232
query57	1202	1126	1144	1126
query58	239	207	215	207
query59	2790	2700	2699	2699
query60	269	337	276	276
query61	116	117	114	114
query62	678	453	440	440
query63	317	291	285	285
query64	5673	4160	4008	4008
query65	3064	3021	3019	3019
query66	903	374	378	374
query67	15077	15270	14755	14755
query68	5626	522	517	517
query69	594	376	384	376
query70	1250	1144	1140	1140
query71	453	282	284	282
query72	6535	2868	2632	2632
query73	710	316	310	310
query74	7195	6671	6686	6671
query75	3673	2850	2904	2850
query76	3808	832	846	832
query77	674	281	260	260
query78	10907	10245	10229	10229
query79	8509	531	515	515
query80	1963	390	416	390
query81	569	215	222	215
query82	1630	211	207	207
query83	288	144	163	144
query84	308	80	83	80
query85	1985	325	316	316
query86	489	298	312	298
query87	3713	3525	3544	3525
query88	5237	2302	2294	2294
query89	532	380	369	369
query90	2053	179	177	177
query91	174	132	138	132
query92	64	53	49	49
query93	6974	492	484	484
query94	1203	186	187	186
query95	433	331	334	331
query96	593	267	267	267
query97	3084	2910	2877	2877
query98	227	208	205	205
query99	1196	908	886	886
Total cold run time: 313042 ms
Total hot run time: 187520 ms

doris-robot · 2024-03-23T05:04:09Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit f14d53e24f2c3c21940587d5dd726642c7affffe with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

liaoxin01 · 2024-03-23T05:54:10Z

run buildall

github-actions · 2024-03-23T05:54:14Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-23T05:58:12Z

clang-tidy review says "All clean, LGTM! 👍"

liaoxin01 · 2024-03-23T05:58:29Z

run buildall

github-actions · 2024-03-23T06:01:50Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-23T06:03:00Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-03-23T06:14:58Z

TeamCity be ut coverage result:
Function Coverage: 35.26% (8733/24766)
Line Coverage: 27.06% (71510/264296)
Region Coverage: 26.29% (37096/141096)
Branch Coverage: 23.19% (18975/81816)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b18873c329e9a0d5419e6890a58b249294a34d8c_b18873c329e9a0d5419e6890a58b249294a34d8c/report/index.html

doris-robot · 2024-03-23T06:27:04Z

TPC-H: Total hot run time: 37876 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b18873c329e9a0d5419e6890a58b249294a34d8c, data reload: false

------ Round 1 ----------------------------------
q1	17645	4334	4088	4088
q2	2096	161	150	150
q3	10597	1091	1189	1091
q4	10231	832	788	788
q5	7452	3021	2965	2965
q6	201	126	127	126
q7	1016	585	587	585
q8	9332	1955	1947	1947
q9	7194	6582	6519	6519
q10	8462	3445	3503	3445
q11	430	231	230	230
q12	401	203	203	203
q13	17787	2887	2865	2865
q14	226	202	204	202
q15	506	460	452	452
q16	490	380	374	374
q17	966	643	589	589
q18	7099	6424	6425	6424
q19	1547	1419	1488	1419
q20	539	255	256	255
q21	3543	2884	2866	2866
q22	357	293	293	293
Total cold run time: 108117 ms
Total hot run time: 37876 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4182	4091	4058	4058
q2	323	236	229	229
q3	2947	2834	2844	2834
q4	1828	1523	1525	1523
q5	5283	5352	5345	5345
q6	195	116	119	116
q7	2223	1847	1827	1827
q8	3137	3275	3275	3275
q9	8659	8694	8702	8694
q10	3750	3783	3762	3762
q11	543	449	440	440
q12	709	531	554	531
q13	16923	2867	2845	2845
q14	277	246	254	246
q15	494	474	452	452
q16	461	425	423	423
q17	1731	1480	1472	1472
q18	7408	7238	7056	7056
q19	1613	1522	1512	1512
q20	1905	1727	1711	1711
q21	4844	4725	4787	4725
q22	527	442	435	435
Total cold run time: 69962 ms
Total hot run time: 53511 ms

doris-robot · 2024-03-23T06:39:15Z

TPC-DS: Total hot run time: 185755 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b18873c329e9a0d5419e6890a58b249294a34d8c, data reload: false

query1	940	357	346	346
query2	7420	2013	1910	1910
query3	6713	210	214	210
query4	31880	21209	21343	21209
query5	4312	410	401	401
query6	256	181	169	169
query7	4634	293	284	284
query8	225	177	179	177
query9	9392	2271	2298	2271
query10	560	261	259	259
query11	17500	14364	14491	14364
query12	137	95	88	88
query13	1626	400	430	400
query14	14158	11060	10593	10593
query15	272	205	197	197
query16	8000	263	258	258
query17	1954	595	549	549
query18	2055	291	295	291
query19	238	159	159	159
query20	92	88	89	88
query21	196	129	128	128
query22	4953	4809	4861	4809
query23	33197	32806	32457	32457
query24	10944	2881	2880	2880
query25	607	379	383	379
query26	984	160	160	160
query27	2373	350	362	350
query28	6893	1886	1841	1841
query29	888	676	629	629
query30	297	150	152	150
query31	991	742	726	726
query32	97	59	58	58
query33	778	302	277	277
query34	1010	487	495	487
query35	830	620	634	620
query36	994	901	902	901
query37	114	77	81	77
query38	3574	3431	3393	3393
query39	1476	1438	1444	1438
query40	214	122	121	121
query41	51	49	51	49
query42	103	102	98	98
query43	496	458	446	446
query44	1165	717	732	717
query45	280	253	262	253
query46	1134	752	718	718
query47	1882	1835	1820	1820
query48	453	356	352	352
query49	1116	337	333	333
query50	758	385	373	373
query51	6714	6646	6668	6646
query52	111	95	92	92
query53	344	280	279	279
query54	332	251	249	249
query55	84	81	78	78
query56	261	228	236	228
query57	1202	1117	1143	1117
query58	237	210	212	210
query59	2773	2639	2653	2639
query60	363	260	258	258
query61	116	114	115	114
query62	677	458	455	455
query63	316	291	283	283
query64	5770	4088	3907	3907
query65	3058	3015	3039	3015
query66	864	387	364	364
query67	15295	14881	14761	14761
query68	10016	532	520	520
query69	676	365	373	365
query70	1372	1139	1173	1139
query71	538	280	276	276
query72	7837	2696	2554	2554
query73	1598	322	317	317
query74	7333	6649	6732	6649
query75	5292	2857	2885	2857
query76	5923	905	944	905
query77	656	267	264	264
query78	10696	10241	9986	9986
query79	11405	519	531	519
query80	1350	401	391	391
query81	507	211	213	211
query82	343	204	203	203
query83	221	152	143	143
query84	283	78	82	78
query85	1011	322	312	312
query86	360	292	300	292
query87	3846	3548	3565	3548
query88	5077	2271	2272	2271
query89	493	377	376	376
query90	2000	178	175	175
query91	167	135	136	135
query92	58	48	49	48
query93	5900	503	482	482
query94	1353	184	174	174
query95	443	341	334	334
query96	604	267	269	267
query97	3050	2916	2890	2890
query98	222	222	219	219
query99	1079	897	936	897
Total cold run time: 321438 ms
Total hot run time: 185755 ms

doris-robot · 2024-03-23T06:42:27Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit b18873c329e9a0d5419e6890a58b249294a34d8c with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       20.4 seconds inserted 10000000 Rows, about 490K ops/s

wm1581066

LGTM

github-actions · 2024-03-23T13:38:10Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

github-actions · 2024-03-23T14:22:15Z

PR approved by at least one committer and no changes requested.

liaoxin01 requested a review from zhangstar333 March 23, 2024 03:49

liaoxin01 force-pushed the fix_timeout branch from 4ce0caf to f14d53e Compare March 23, 2024 04:30

liaoxin01 force-pushed the fix_timeout branch 2 times, most recently from bed8aa2 to ecc9101 Compare March 23, 2024 05:53

liaoxin01 force-pushed the fix_timeout branch from ecc9101 to 7606623 Compare March 23, 2024 05:56

[fix](move-memtable) fix load timeout caused by lost wakeup

b18873c

liaoxin01 force-pushed the fix_timeout branch from 7606623 to b18873c Compare March 23, 2024 05:58

wm1581066 approved these changes Mar 23, 2024

View reviewed changes

github-actions bot added the reviewed label Mar 23, 2024

dataroaring approved these changes Mar 23, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 23, 2024

dataroaring merged commit 9312469 into apache:master Mar 23, 2024

yiguolei pushed a commit that referenced this pull request Mar 24, 2024

[fix](move-memtable) fix load timeout caused by lost wakeup (#32720)

5c3ca0f

[fix](move-memtable) fix load timeout caused by lost wakeup #32720

[fix](move-memtable) fix load timeout caused by lost wakeup #32720

Uh oh!

Conversation

liaoxin01 commented Mar 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented Mar 23, 2024

Uh oh!

liaoxin01 commented Mar 23, 2024

Uh oh!

github-actions bot commented Mar 23, 2024

Uh oh!

doris-robot commented Mar 23, 2024

Uh oh!

doris-robot commented Mar 23, 2024

Uh oh!

doris-robot commented Mar 23, 2024

Uh oh!

doris-robot commented Mar 23, 2024

Uh oh!

liaoxin01 commented Mar 23, 2024

Uh oh!

github-actions bot commented Mar 23, 2024

Uh oh!

doris-robot commented Mar 23, 2024

Uh oh!

doris-robot commented Mar 23, 2024

Uh oh!

doris-robot commented Mar 23, 2024

Uh oh!

doris-robot commented Mar 23, 2024

Uh oh!

liaoxin01 commented Mar 23, 2024

Uh oh!

github-actions bot commented Mar 23, 2024

Uh oh!

github-actions bot commented Mar 23, 2024

Uh oh!

liaoxin01 commented Mar 23, 2024

Uh oh!

github-actions bot commented Mar 23, 2024

Uh oh!

github-actions bot commented Mar 23, 2024

Uh oh!

doris-robot commented Mar 23, 2024

Uh oh!

doris-robot commented Mar 23, 2024

Uh oh!

doris-robot commented Mar 23, 2024

Uh oh!

doris-robot commented Mar 23, 2024

Uh oh!

wm1581066 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 23, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

liaoxin01 commented Mar 23, 2024 •

edited

Loading