[fix](routine-load) avoid routine load pause for check transaction status fail #32638

sollhui · 2024-03-21T12:32:41Z

Proposed changes

routine load will pause in follow case:

routine load task visible and changed transaction status
user show task which cause task transaction status changed to visible
routineLoadTaskInfo.getTxnStatus() != TransactionStatus.COMMITTED check fail

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-03-21T12:32:45Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

sollhui · 2024-03-21T12:32:53Z

run buildall

doris-robot · 2024-03-21T12:52:47Z

TPC-H: Total hot run time: 38424 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 64b900cfaba6bf9c7c694d465e16b97c2e9f5e82, data reload: false

------ Round 1 ----------------------------------
q1	17844	4936	4263	4263
q2	2467	160	152	152
q3	11524	1188	1208	1188
q4	10280	787	797	787
q5	7525	3042	3046	3042
q6	210	130	127	127
q7	1063	648	596	596
q8	10283	2063	2009	2009
q9	7246	6655	6578	6578
q10	9164	3466	3527	3466
q11	431	227	214	214
q12	397	212	199	199
q13	17810	2868	2902	2868
q14	233	199	208	199
q15	499	461	456	456
q16	467	362	360	360
q17	972	618	564	564
q18	7250	6633	6541	6541
q19	1582	1423	1495	1423
q20	538	271	245	245
q21	3654	3009	2848	2848
q22	350	315	299	299
Total cold run time: 111789 ms
Total hot run time: 38424 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4176	4085	4072	4072
q2	335	228	232	228
q3	3033	2846	2798	2798
q4	1842	1568	1594	1568
q5	5268	5304	5276	5276
q6	202	118	118	118
q7	2248	1890	1838	1838
q8	3169	3301	3335	3301
q9	8612	8577	8608	8577
q10	3710	3708	3662	3662
q11	543	437	435	435
q12	726	557	571	557
q13	16902	2887	2884	2884
q14	273	246	247	246
q15	492	443	451	443
q16	462	408	425	408
q17	1767	1508	1473	1473
q18	7519	7289	7155	7155
q19	1596	1537	1541	1537
q20	1913	1708	1707	1707
q21	4862	4601	4917	4601
q22	519	440	452	440
Total cold run time: 70169 ms
Total hot run time: 53324 ms

doris-robot · 2024-03-21T13:04:42Z

TPC-DS: Total hot run time: 180827 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 64b900cfaba6bf9c7c694d465e16b97c2e9f5e82, data reload: false

query1	936	369	338	338
query2	7482	2008	2042	2008
query3	6709	213	221	213
query4	31198	20609	20649	20609
query5	4366	404	410	404
query6	277	191	176	176
query7	4626	284	293	284
query8	226	165	170	165
query9	9990	2295	2292	2292
query10	486	239	248	239
query11	17095	14158	13995	13995
query12	148	88	87	87
query13	1638	421	427	421
query14	11482	10580	10345	10345
query15	255	204	192	192
query16	8096	263	265	263
query17	1935	585	551	551
query18	2054	280	271	271
query19	215	154	155	154
query20	89	88	90	88
query21	197	129	123	123
query22	4680	4465	4437	4437
query23	31626	30927	31180	30927
query24	10851	2788	2814	2788
query25	627	367	361	361
query26	1446	147	153	147
query27	3160	346	346	346
query28	7965	1930	1910	1910
query29	877	631	595	595
query30	295	138	138	138
query31	967	727	718	718
query32	102	62	55	55
query33	785	258	277	258
query34	1066	482	479	479
query35	840	628	609	609
query36	1016	839	837	837
query37	125	80	79	79
query38	3478	3424	3409	3409
query39	1418	1386	1407	1386
query40	208	117	110	110
query41	49	56	46	46
query42	107	95	95	95
query43	492	452	439	439
query44	1329	738	710	710
query45	275	272	270	270
query46	1112	682	674	674
query47	1667	1586	1623	1586
query48	456	355	357	355
query49	1145	340	329	329
query50	771	378	373	373
query51	6575	6566	6645	6566
query52	113	89	96	89
query53	347	273	276	273
query54	312	253	251	251
query55	83	80	80	80
query56	260	247	229	229
query57	1083	1005	1029	1005
query58	237	206	216	206
query59	2748	2618	2726	2618
query60	272	253	291	253
query61	117	112	109	109
query62	613	405	398	398
query63	305	280	275	275
query64	5457	3955	4003	3955
query65	3103	2981	2985	2981
query66	1460	357	352	352
query67	14982	14536	14449	14449
query68	6805	515	527	515
query69	614	380	378	378
query70	1263	1181	1188	1181
query71	511	291	284	284
query72	6905	2685	2490	2490
query73	734	311	318	311
query74	7482	6573	6596	6573
query75	4217	2821	2877	2821
query76	4774	945	1015	945
query77	641	265	258	258
query78	10440	9606	9645	9606
query79	12145	533	520	520
query80	1754	397	384	384
query81	537	211	209	209
query82	1370	197	204	197
query83	206	144	142	142
query84	285	80	79	79
query85	1520	331	319	319
query86	467	304	306	304
query87	3785	3523	3522	3522
query88	5475	2283	2267	2267
query89	517	362	362	362
query90	1986	178	173	173
query91	166	134	133	133
query92	57	46	48	46
query93	7683	493	474	474
query94	1252	175	172	172
query95	420	331	322	322
query96	599	267	265	265
query97	3061	2847	2890	2847
query98	227	214	212	212
query99	1261	743	755	743
Total cold run time: 315661 ms
Total hot run time: 180827 ms

doris-robot · 2024-03-21T13:07:57Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 64b900cfaba6bf9c7c694d465e16b97c2e9f5e82 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

liaoxin01

LGTM

github-actions · 2024-03-21T13:11:44Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-03-21T13:11:45Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

dataroaring

The condition is always true?

sollhui · 2024-03-21T14:51:24Z

run buildall

dataroaring

LGTM

github-actions · 2024-03-21T14:54:36Z

PR approved by at least one committer and no changes requested.

doris-robot · 2024-03-21T15:49:10Z

TPC-H: Total hot run time: 37668 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1624763bb330c7be8e7e3150239211d05d999938, data reload: false

------ Round 1 ----------------------------------
q1	17656	4184	4132	4132
q2	2023	149	155	149
q3	10595	1133	1197	1133
q4	10224	836	785	785
q5	7454	2967	2937	2937
q6	200	121	120	120
q7	1026	572	583	572
q8	9338	2041	1975	1975
q9	7148	6494	6453	6453
q10	8416	3414	3545	3414
q11	447	221	219	219
q12	408	193	195	193
q13	17799	2850	2868	2850
q14	239	201	205	201
q15	501	460	459	459
q16	449	358	353	353
q17	961	526	583	526
q18	7201	6497	6401	6401
q19	1658	1431	1457	1431
q20	551	258	243	243
q21	3559	2898	2818	2818
q22	334	304	335	304
Total cold run time: 108187 ms
Total hot run time: 37668 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4139	4132	4143	4132
q2	320	219	224	219
q3	3010	2875	2868	2868
q4	1796	1486	1554	1486
q5	5244	5234	5259	5234
q6	192	114	115	114
q7	2214	1895	1874	1874
q8	3167	3319	3288	3288
q9	8571	8572	8557	8557
q10	3709	3680	3730	3680
q11	542	436	435	435
q12	721	551	557	551
q13	16930	2828	2883	2828
q14	278	255	255	255
q15	485	445	437	437
q16	446	416	426	416
q17	1735	1478	1453	1453
q18	7598	7192	7178	7178
q19	1630	1544	1512	1512
q20	1907	1712	1701	1701
q21	4908	4631	4711	4631
q22	503	466	470	466
Total cold run time: 70045 ms
Total hot run time: 53315 ms

doris-robot · 2024-03-21T16:01:01Z

TPC-DS: Total hot run time: 180704 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1624763bb330c7be8e7e3150239211d05d999938, data reload: false

query1	918	364	332	332
query2	7391	2023	2013	2013
query3	6701	208	210	208
query4	31166	20668	20698	20668
query5	4250	417	388	388
query6	265	167	170	167
query7	4629	286	286	286
query8	230	157	170	157
query9	9831	2270	2260	2260
query10	473	240	241	240
query11	17187	14128	14133	14128
query12	136	97	90	90
query13	1630	412	412	412
query14	11763	10427	10344	10344
query15	255	205	187	187
query16	8057	251	246	246
query17	1963	546	513	513
query18	2043	267	270	267
query19	248	158	149	149
query20	93	87	87	87
query21	198	130	126	126
query22	4704	4462	4387	4387
query23	31810	31021	31214	31021
query24	10775	2782	2880	2782
query25	607	380	384	380
query26	993	154	158	154
query27	2379	347	337	337
query28	7148	1884	1834	1834
query29	864	655	616	616
query30	307	144	143	143
query31	967	723	731	723
query32	89	58	55	55
query33	764	254	256	254
query34	1083	460	469	460
query35	834	601	599	599
query36	982	856	903	856
query37	117	73	77	73
query38	3570	3513	3478	3478
query39	1422	1359	1363	1359
query40	214	116	112	112
query41	51	46	46	46
query42	106	95	99	95
query43	489	443	465	443
query44	1236	701	699	699
query45	260	265	259	259
query46	1075	694	681	681
query47	1661	1620	1576	1576
query48	426	357	351	351
query49	1115	355	343	343
query50	757	369	370	369
query51	6703	6616	6585	6585
query52	110	93	93	93
query53	337	268	273	268
query54	314	241	250	241
query55	88	82	79	79
query56	244	306	232	232
query57	1071	1004	1001	1001
query58	239	205	206	205
query59	2658	2629	2575	2575
query60	294	256	247	247
query61	114	113	109	109
query62	612	420	421	420
query63	297	274	272	272
query64	5727	3828	3923	3828
query65	3027	2982	3000	2982
query66	869	361	347	347
query67	14881	14518	14498	14498
query68	9086	532	532	532
query69	614	391	368	368
query70	1433	1179	1127	1127
query71	521	281	275	275
query72	6338	2665	2492	2492
query73	1615	305	312	305
query74	7623	6576	6591	6576
query75	4414	2910	2867	2867
query76	5563	881	872	872
query77	676	267	251	251
query78	10259	9675	9655	9655
query79	11435	525	525	525
query80	2363	381	370	370
query81	490	209	210	209
query82	351	190	194	190
query83	254	140	142	140
query84	290	80	77	77
query85	1114	323	306	306
query86	361	304	282	282
query87	3763	3511	3526	3511
query88	5520	2288	2275	2275
query89	458	357	359	357
query90	2045	172	174	172
query91	166	131	135	131
query92	60	46	47	46
query93	5953	504	482	482
query94	1345	177	174	174
query95	421	316	324	316
query96	623	270	265	265
query97	3046	2869	2893	2869
query98	214	215	209	209
query99	946	773	743	743
Total cold run time: 313177 ms
Total hot run time: 180704 ms

doris-robot · 2024-03-21T16:04:13Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 1624763bb330c7be8e7e3150239211d05d999938 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.2 seconds inserted 10000000 Rows, about 471K ops/s

cambyzju

LGTM

…atus fail (#32638)

…atus fail (#32638) (#37444) pick #32638 Co-authored-by: HHoflittlefish777 <77738092+HHoflittlefish777@users.noreply.github.com>

…atus fail (apache#32638) (apache#37444) pick apache#32638 Co-authored-by: HHoflittlefish777 <77738092+HHoflittlefish777@users.noreply.github.com>

liaoxin01 previously approved these changes Mar 21, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 21, 2024

github-actions bot added the reviewed label Mar 21, 2024

dataroaring approved these changes Mar 21, 2024

View reviewed changes

dataroaring requested changes Mar 21, 2024

View reviewed changes

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 21, 2024

avoid routine load pause for check transaction status fail

1624763

sollhui dismissed liaoxin01’s stale review via 1624763 March 21, 2024 14:51

dataroaring approved these changes Mar 21, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 21, 2024

cambyzju approved these changes Mar 22, 2024

View reviewed changes

liaoxin01 merged commit 7bb1d79 into apache:master Mar 22, 2024

yiguolei added the 2.1.0-conflict label Mar 22, 2024

yiguolei pushed a commit that referenced this pull request Mar 22, 2024

[fix](routine-load) avoid routine load pause for check transaction st…

8a6fc79

…atus fail (#32638)

xiaokang added the dev/2.0.x label Mar 22, 2024

yiguolei mentioned this pull request Mar 24, 2024

Release notes 2.1.1 #32736

Closed

yiguolei added dev/2.1.1-merged and removed 2.1.0-conflict labels Mar 25, 2024

sollhui mentioned this pull request Jul 8, 2024

[fix](routine-load) avoid routine load pause for check transaction status fail (#32638) #37444

Merged

dataroaring pushed a commit that referenced this pull request Jul 9, 2024

[fix](routine-load) avoid routine load pause for check transaction st…

f96de8e

…atus fail (#32638) (#37444) pick #32638 Co-authored-by: HHoflittlefish777 <77738092+HHoflittlefish777@users.noreply.github.com>

dataroaring added dev/2.0.13-merged and removed dev/2.0.x labels Jul 9, 2024

[fix](routine-load) avoid routine load pause for check transaction status fail #32638

[fix](routine-load) avoid routine load pause for check transaction status fail #32638

Uh oh!

Conversation

sollhui commented Mar 21, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented Mar 21, 2024

Uh oh!

sollhui commented Mar 21, 2024

Uh oh!

doris-robot commented Mar 21, 2024

Uh oh!

doris-robot commented Mar 21, 2024

Uh oh!

doris-robot commented Mar 21, 2024

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 21, 2024

Uh oh!

github-actions bot commented Mar 21, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

sollhui commented Mar 21, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 21, 2024

Uh oh!

doris-robot commented Mar 21, 2024

Uh oh!

doris-robot commented Mar 21, 2024

Uh oh!

doris-robot commented Mar 21, 2024

Uh oh!

cambyzju left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants