[fix](routine-load) fix timeout backoff can not work #32661

sollhui · 2024-03-22T03:31:10Z

Proposed changes

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-03-22T03:31:14Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

sollhui · 2024-03-22T03:31:22Z

run buildall

doris-robot · 2024-03-22T03:58:18Z

TPC-H: Total hot run time: 37329 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8cc31635e9be0e95f0233ef98d521b05b189a472, data reload: false

------ Round 1 ----------------------------------
q1	17662	4193	4091	4091
q2	2025	151	143	143
q3	10599	1121	1178	1121
q4	10228	794	760	760
q5	7465	2921	2919	2919
q6	205	123	120	120
q7	1016	589	566	566
q8	9335	2010	1953	1953
q9	7274	6445	6404	6404
q10	8388	3324	3464	3324
q11	425	220	222	220
q12	397	200	191	191
q13	17791	2863	2842	2842
q14	240	202	220	202
q15	527	445	445	445
q16	477	364	357	357
q17	935	575	553	553
q18	7087	6402	6338	6338
q19	3200	1416	1469	1416
q20	543	265	261	261
q21	3552	2879	2802	2802
q22	355	301	302	301
Total cold run time: 109726 ms
Total hot run time: 37329 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4119	4077	4044	4044
q2	325	222	230	222
q3	2917	2788	2814	2788
q4	1856	1576	1530	1530
q5	5203	5241	5219	5219
q6	192	117	117	117
q7	2231	1826	1809	1809
q8	3140	3262	3271	3262
q9	8521	8523	8543	8523
q10	3706	3704	3688	3688
q11	542	448	439	439
q12	699	581	573	573
q13	16929	2847	2826	2826
q14	277	259	257	257
q15	488	460	474	460
q16	462	412	401	401
q17	1721	1495	1464	1464
q18	7486	7069	7155	7069
q19	2404	1511	1521	1511
q20	1957	1700	1681	1681
q21	4905	4581	4519	4519
q22	504	437	436	436
Total cold run time: 70584 ms
Total hot run time: 52838 ms

doris-robot · 2024-03-22T04:10:09Z

TPC-DS: Total hot run time: 181151 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8cc31635e9be0e95f0233ef98d521b05b189a472, data reload: false

query1	933	358	341	341
query2	7402	1960	1854	1854
query3	6713	210	211	210
query4	30991	20612	20595	20595
query5	4323	403	393	393
query6	264	176	179	176
query7	4633	285	302	285
query8	228	172	164	164
query9	9112	2290	2272	2272
query10	458	245	243	243
query11	17018	14358	14236	14236
query12	144	90	86	86
query13	1628	416	421	416
query14	10803	11445	10561	10561
query15	280	208	192	192
query16	8165	265	254	254
query17	1948	573	541	541
query18	2099	292	287	287
query19	318	165	157	157
query20	101	88	85	85
query21	203	128	124	124
query22	4703	4602	4451	4451
query23	31726	31118	31173	31118
query24	10641	2839	2798	2798
query25	580	388	383	383
query26	718	161	157	157
query27	2225	349	351	349
query28	6116	1920	1882	1882
query29	872	675	643	643
query30	299	148	149	148
query31	969	744	718	718
query32	101	58	60	58
query33	776	260	249	249
query34	1069	465	463	463
query35	805	602	600	600
query36	973	861	852	852
query37	106	74	75	74
query38	3544	3380	3429	3380
query39	1435	1365	1385	1365
query40	211	115	113	113
query41	55	47	48	47
query42	102	97	100	97
query43	459	445	451	445
query44	1187	724	723	723
query45	265	257	263	257
query46	1088	705	700	700
query47	1700	1621	1598	1598
query48	436	359	351	351
query49	1100	352	340	340
query50	761	368	367	367
query51	6800	6611	6586	6586
query52	113	87	95	87
query53	341	283	275	275
query54	313	241	242	241
query55	91	80	81	80
query56	260	232	234	232
query57	1079	1015	1020	1015
query58	226	206	204	204
query59	2856	2667	2581	2581
query60	275	234	256	234
query61	114	112	114	112
query62	626	410	397	397
query63	299	276	271	271
query64	5481	3884	3908	3884
query65	3018	2998	2995	2995
query66	858	335	333	333
query67	15020	14508	14341	14341
query68	9024	521	528	521
query69	653	371	371	371
query70	1316	1164	1137	1137
query71	515	282	267	267
query72	6661	2666	2512	2512
query73	1605	316	317	316
query74	8434	6605	6658	6605
query75	4521	2763	2845	2763
query76	5689	935	925	925
query77	670	256	267	256
query78	10288	9826	9648	9648
query79	11527	514	513	513
query80	1812	403	385	385
query81	518	216	214	214
query82	339	209	201	201
query83	212	148	145	145
query84	295	78	77	77
query85	1000	323	309	309
query86	351	306	305	305
query87	3704	3537	3572	3537
query88	5447	2306	2291	2291
query89	464	359	354	354
query90	2033	178	175	175
query91	173	155	142	142
query92	63	46	47	46
query93	6169	495	482	482
query94	1333	174	178	174
query95	440	331	324	324
query96	610	270	269	269
query97	3026	2861	2874	2861
query98	236	205	211	205
query99	1003	755	766	755
Total cold run time: 310687 ms
Total hot run time: 181151 ms

doris-robot · 2024-03-22T04:13:24Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 8cc31635e9be0e95f0233ef98d521b05b189a472 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.9 seconds inserted 10000000 Rows, about 456K ops/s

sollhui · 2024-03-22T04:46:29Z

run buildall

doris-robot · 2024-03-22T05:10:28Z

TPC-H: Total hot run time: 38480 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a5ff2eb4520f8709c00fa48196ccf6a5b1989cd5, data reload: false

------ Round 1 ----------------------------------
q1	17643	4653	4183	4183
q2	2026	152	145	145
q3	11227	1158	1235	1158
q4	10557	846	849	846
q5	7562	3065	3036	3036
q6	208	130	127	127
q7	1086	611	596	596
q8	9445	2045	2012	2012
q9	7313	6635	6625	6625
q10	8858	3546	3632	3546
q11	449	227	215	215
q12	595	206	193	193
q13	19989	2863	2906	2863
q14	222	201	203	201
q15	504	457	466	457
q16	443	365	358	358
q17	957	515	597	515
q18	7282	6600	6738	6600
q19	3029	1455	1505	1455
q20	551	284	251	251
q21	3580	2992	2797	2797
q22	354	301	303	301
Total cold run time: 113880 ms
Total hot run time: 38480 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4120	4130	4107	4107
q2	329	223	225	223
q3	3017	2859	2810	2810
q4	1893	1579	1578	1578
q5	5242	5305	5300	5300
q6	197	115	119	115
q7	2240	1838	1869	1838
q8	3199	3328	3316	3316
q9	8616	8643	8550	8550
q10	3730	3682	3693	3682
q11	538	448	438	438
q12	736	560	547	547
q13	16919	2864	2838	2838
q14	280	246	249	246
q15	493	465	463	463
q16	465	423	418	418
q17	1759	1481	1502	1481
q18	7446	7297	7158	7158
q19	1627	1565	1565	1565
q20	1894	1729	1689	1689
q21	4940	4635	4678	4635
q22	544	442	459	442
Total cold run time: 70224 ms
Total hot run time: 53439 ms

doris-robot · 2024-03-22T05:22:32Z

TPC-DS: Total hot run time: 181820 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a5ff2eb4520f8709c00fa48196ccf6a5b1989cd5, data reload: false

query1	923	373	366	366
query2	7346	1995	1957	1957
query3	6709	211	219	211
query4	31609	20802	20623	20623
query5	4344	422	410	410
query6	281	171	175	171
query7	4632	296	292	292
query8	238	176	170	170
query9	9294	2301	2285	2285
query10	479	241	246	241
query11	17149	14105	14189	14105
query12	139	91	85	85
query13	1620	432	415	415
query14	14453	10993	10955	10955
query15	264	199	191	191
query16	8250	273	257	257
query17	1988	582	545	545
query18	2104	283	285	283
query19	367	150	161	150
query20	92	89	90	89
query21	202	127	131	127
query22	4838	4611	4595	4595
query23	31730	31074	31103	31074
query24	12053	2877	2828	2828
query25	650	374	380	374
query26	1794	150	156	150
query27	2967	364	352	352
query28	7732	1910	1864	1864
query29	1022	641	647	641
query30	309	150	147	147
query31	957	731	731	731
query32	99	59	57	57
query33	772	258	334	258
query34	914	477	469	469
query35	835	615	599	599
query36	968	861	897	861
query37	274	78	76	76
query38	3561	3483	3448	3448
query39	1396	1374	1369	1369
query40	296	112	111	111
query41	52	47	48	47
query42	105	95	99	95
query43	458	459	461	459
query44	1179	718	727	718
query45	270	264	268	264
query46	1073	695	700	695
query47	1676	1615	1573	1573
query48	444	369	360	360
query49	1232	329	330	329
query50	759	368	375	368
query51	6581	6596	6573	6573
query52	105	90	92	90
query53	344	278	274	274
query54	340	254	245	245
query55	89	79	82	79
query56	275	255	245	245
query57	1109	1012	1030	1012
query58	246	207	210	207
query59	2836	2615	2588	2588
query60	276	245	260	245
query61	115	112	121	112
query62	594	432	407	407
query63	305	271	273	271
query64	6826	3808	3744	3744
query65	3091	2999	3002	2999
query66	1413	351	328	328
query67	15400	14600	14641	14600
query68	8875	510	536	510
query69	615	389	393	389
query70	1285	1094	1173	1094
query71	510	274	284	274
query72	6332	2672	2488	2488
query73	770	318	323	318
query74	7020	6549	6582	6549
query75	4089	2828	2847	2828
query76	5377	867	833	833
query77	651	255	256	255
query78	10481	9635	9651	9635
query79	12742	514	530	514
query80	2757	386	386	386
query81	506	221	209	209
query82	661	200	197	197
query83	249	141	147	141
query84	284	78	77	77
query85	1231	332	311	311
query86	354	316	294	294
query87	3762	3568	3554	3554
query88	5127	2301	2286	2286
query89	496	373	373	373
query90	2044	174	178	174
query91	169	133	136	133
query92	66	46	51	46
query93	7445	495	479	479
query94	1381	179	184	179
query95	427	331	330	330
query96	614	275	268	268
query97	3036	2907	2888	2888
query98	223	209	212	209
query99	1014	747	740	740
Total cold run time: 323436 ms
Total hot run time: 181820 ms

doris-robot · 2024-03-22T05:25:46Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a5ff2eb4520f8709c00fa48196ccf6a5b1989cd5 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.7 seconds inserted 10000000 Rows, about 460K ops/s

liaoxin01

LGTM

github-actions · 2024-03-22T08:16:36Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-03-22T08:16:38Z

PR approved by anyone and no changes requested.

XuJianxu

LGTM

…32661)" This reverts commit 0d0f787.

#32709) This reverts commit 0d0f787. Co-authored-by: stephen <hello-stephen@qq.com>

* [opt](routine-load) self-adaption backoff timeout (#32227) * [fix](routine-load) fix timeout backoff can not work (#32661)

fix timeout backoff can not work

a5ff2eb

liaoxin01 approved these changes Mar 22, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 22, 2024

github-actions bot added the reviewed label Mar 22, 2024

XuJianxu approved these changes Mar 22, 2024

View reviewed changes

liaoxin01 merged commit d536ad1 into apache:master Mar 22, 2024

yiguolei pushed a commit that referenced this pull request Mar 22, 2024

[fix](routine-load) fix timeout backoff can not work (#32661)

0d0f787

xiaokang added the dev/2.0.x label Mar 22, 2024

yiguolei added the 2.1.0-conflict label Mar 22, 2024

hello-stephen pushed a commit to hello-stephen/doris that referenced this pull request Mar 22, 2024

Revert "[fix](routine-load) fix timeout backoff can not work (apache#…

c9b0ea3

…32661)" This reverts commit 0d0f787.

yiguolei pushed a commit that referenced this pull request Mar 22, 2024

Revert "[fix](routine-load) fix timeout backoff can not work (#32661)" (

8514fab

#32709) This reverts commit 0d0f787. Co-authored-by: stephen <hello-stephen@qq.com>

yiguolei pushed a commit that referenced this pull request Mar 24, 2024

[branch-2.1](routine-load) self-adaption backoff timeout (#32734)

1b0cd4a

* [opt](routine-load) self-adaption backoff timeout (#32227) * [fix](routine-load) fix timeout backoff can not work (#32661)

yiguolei added dev/2.1.1-merged and removed 2.1.0-conflict labels Mar 25, 2024

liaoxin01 removed the dev/2.0.x label Jul 8, 2024

[fix](routine-load) fix timeout backoff can not work #32661

[fix](routine-load) fix timeout backoff can not work #32661

Uh oh!

Conversation

sollhui commented Mar 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented Mar 22, 2024

Uh oh!

sollhui commented Mar 22, 2024

Uh oh!

doris-robot commented Mar 22, 2024

Uh oh!

doris-robot commented Mar 22, 2024

Uh oh!

doris-robot commented Mar 22, 2024

Uh oh!

sollhui commented Mar 22, 2024

Uh oh!

doris-robot commented Mar 22, 2024

Uh oh!

doris-robot commented Mar 22, 2024

Uh oh!

doris-robot commented Mar 22, 2024

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 22, 2024

Uh oh!

github-actions bot commented Mar 22, 2024

Uh oh!

XuJianxu left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

sollhui commented Mar 22, 2024 •

edited

Loading