[fix](insert) Insert init plan error may forget to abort txn #34220

mymeiyi · 2024-04-28T03:41:14Z

Proposed changes

When FE generate plan for insert into command:

begin txn and finalize sink
if finalize sink throws exception, the txn is not aborted

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-04-28T03:41:19Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

morningman

LGTM

mymeiyi · 2024-04-28T03:48:43Z

run buildall

github-actions · 2024-04-28T03:49:02Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-04-28T03:49:03Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

doris-robot · 2024-04-28T04:07:04Z

TPC-H: Total hot run time: 40179 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 955cc76783988f34fa4bd9022f0b6cf3461be15e, data reload: false

------ Round 1 ----------------------------------
q1	17814	4434	4382	4382
q2	2452	199	209	199
q3	10889	1279	1237	1237
q4	11089	724	813	724
q5	7517	2698	2703	2698
q6	213	127	129	127
q7	999	616	592	592
q8	9236	2118	2064	2064
q9	8982	6607	6568	6568
q10	8626	3754	3712	3712
q11	456	241	236	236
q12	393	222	217	217
q13	17746	2953	2949	2949
q14	265	220	227	220
q15	501	466	464	464
q16	489	384	378	378
q17	963	671	692	671
q18	8037	7645	7400	7400
q19	1631	1540	1527	1527
q20	677	301	299	299
q21	4940	3242	4053	3242
q22	344	273	291	273
Total cold run time: 114259 ms
Total hot run time: 40179 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4241	4179	4171	4171
q2	366	265	278	265
q3	2989	2764	2748	2748
q4	1891	1560	1584	1560
q5	5275	5258	5293	5258
q6	209	123	121	121
q7	2239	1872	1824	1824
q8	3177	3326	3356	3326
q9	8526	8520	8567	8520
q10	3910	3679	3698	3679
q11	572	479	491	479
q12	753	583	606	583
q13	17062	2941	2948	2941
q14	305	276	271	271
q15	516	472	486	472
q16	462	410	422	410
q17	1750	1491	1453	1453
q18	7538	7572	7252	7252
q19	1621	1552	1554	1552
q20	1949	1751	1775	1751
q21	4793	4807	4758	4758
q22	563	500	497	497
Total cold run time: 70707 ms
Total hot run time: 53891 ms

doris-robot · 2024-04-28T04:18:56Z

TPC-DS: Total hot run time: 185571 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 955cc76783988f34fa4bd9022f0b6cf3461be15e, data reload: false

query1	909	371	340	340
query2	6462	2465	2288	2288
query3	6655	206	217	206
query4	24267	21173	21132	21132
query5	4136	404	428	404
query6	269	187	177	177
query7	4588	292	287	287
query8	250	198	194	194
query9	8409	2339	2304	2304
query10	419	235	250	235
query11	16360	14276	14241	14241
query12	134	87	86	86
query13	1631	356	361	356
query14	10570	8070	7526	7526
query15	256	168	170	168
query16	8181	256	255	255
query17	1853	552	535	535
query18	2099	265	261	261
query19	285	144	146	144
query20	91	86	85	85
query21	199	124	121	121
query22	5124	4815	4791	4791
query23	34183	33029	33097	33029
query24	11636	2932	2907	2907
query25	650	359	361	359
query26	1713	153	149	149
query27	2934	325	339	325
query28	7402	1989	1984	1984
query29	962	605	583	583
query30	290	150	151	150
query31	957	742	738	738
query32	90	50	51	50
query33	742	246	240	240
query34	1085	479	481	479
query35	800	668	649	649
query36	1057	890	924	890
query37	174	66	68	66
query38	3177	3006	3005	3005
query39	1600	1527	1524	1524
query40	277	122	125	122
query41	45	39	39	39
query42	103	100	97	97
query43	571	543	548	543
query44	1245	727	737	727
query45	280	264	263	263
query46	1087	701	697	697
query47	1915	1874	1847	1847
query48	374	287	287	287
query49	1218	388	399	388
query50	766	385	373	373
query51	6883	6853	6784	6784
query52	101	90	94	90
query53	347	288	276	276
query54	314	223	239	223
query55	77	70	71	70
query56	242	233	219	219
query57	1206	1154	1133	1133
query58	226	194	198	194
query59	3495	3063	3072	3063
query60	252	225	233	225
query61	103	87	134	87
query62	637	460	444	444
query63	311	277	276	276
query64	9535	7097	7108	7097
query65	3087	3051	3004	3004
query66	1378	342	322	322
query67	15534	14788	15185	14788
query68	5264	539	542	539
query69	475	306	307	306
query70	1187	1083	1078	1078
query71	400	279	268	268
query72	7216	2754	2514	2514
query73	708	324	324	324
query74	6551	6067	5974	5974
query75	3367	2646	2706	2646
query76	3230	1058	1053	1053
query77	407	267	269	267
query78	10843	10248	10296	10248
query79	8391	524	521	521
query80	2047	459	452	452
query81	544	225	225	225
query82	1558	97	96	96
query83	274	170	169	169
query84	270	90	88	88
query85	2133	326	320	320
query86	486	289	277	277
query87	3271	3097	3077	3077
query88	5311	2311	2313	2311
query89	505	384	371	371
query90	2040	178	185	178
query91	128	97	97	97
query92	61	46	46	46
query93	6405	509	500	500
query94	1239	171	178	171
query95	386	293	288	288
query96	597	261	258	258
query97	3122	2913	2944	2913
query98	246	210	212	210
query99	1295	854	845	845
Total cold run time: 304165 ms
Total hot run time: 185571 ms

mymeiyi · 2024-04-28T07:21:03Z

run buildall

doris-robot · 2024-04-28T08:21:55Z

TPC-H: Total hot run time: 41319 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 05f3cb7d441443ded5e5966d49b4b23ce7290959, data reload: false

------ Round 1 ----------------------------------
q1	17599	4919	4277	4277
q2	2023	191	190	190
q3	10518	1202	1215	1202
q4	10203	861	775	775
q5	7558	2764	2707	2707
q6	230	131	133	131
q7	1052	630	621	621
q8	9225	2163	2135	2135
q9	9256	6802	6884	6802
q10	9026	3899	3964	3899
q11	450	240	244	240
q12	465	240	232	232
q13	18297	3119	3252	3119
q14	265	237	220	220
q15	507	487	463	463
q16	493	406	401	401
q17	980	689	702	689
q18	8435	7872	7779	7779
q19	6275	1570	1522	1522
q20	650	316	318	316
q21	5271	4261	3326	3326
q22	351	273	275	273
Total cold run time: 119129 ms
Total hot run time: 41319 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4527	4357	4478	4357
q2	373	271	264	264
q3	3202	2954	2904	2904
q4	1892	1610	1571	1571
q5	5526	5485	5516	5485
q6	216	121	126	121
q7	2324	1991	1993	1991
q8	3265	3429	3395	3395
q9	8880	8890	8877	8877
q10	3901	3785	3835	3785
q11	585	491	491	491
q12	812	602	658	602
q13	16091	3176	3142	3142
q14	322	298	292	292
q15	504	493	492	492
q16	508	433	436	433
q17	1781	1497	1499	1497
q18	7744	7756	7499	7499
q19	1651	1553	1549	1549
q20	2018	1781	1753	1753
q21	14310	4860	4730	4730
q22	590	509	475	475
Total cold run time: 81022 ms
Total hot run time: 55705 ms

doris-robot · 2024-04-28T08:33:21Z

TPC-DS: Total hot run time: 186820 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 05f3cb7d441443ded5e5966d49b4b23ce7290959, data reload: false

query1	914	371	353	353
query2	6454	2369	2548	2369
query3	6638	209	212	209
query4	23102	21178	21162	21162
query5	4198	420	418	418
query6	266	178	176	176
query7	4587	294	298	294
query8	269	196	182	182
query9	8484	2377	2361	2361
query10	433	246	267	246
query11	14745	14245	14219	14219
query12	135	88	87	87
query13	1647	384	393	384
query14	10534	7590	7891	7590
query15	220	177	169	169
query16	7858	269	262	262
query17	1715	566	537	537
query18	2002	283	282	282
query19	196	159	161	159
query20	97	87	83	83
query21	196	127	121	121
query22	5074	4886	4798	4798
query23	33998	33227	33265	33227
query24	6708	2976	2960	2960
query25	545	370	357	357
query26	698	160	148	148
query27	2081	314	322	314
query28	3638	2058	2044	2044
query29	867	604	586	586
query30	232	160	153	153
query31	970	765	722	722
query32	93	53	52	52
query33	484	269	276	269
query34	870	486	495	486
query35	764	703	665	665
query36	1073	901	962	901
query37	109	68	67	67
query38	3194	3067	3033	3033
query39	1607	1521	1536	1521
query40	203	119	124	119
query41	41	39	38	38
query42	102	92	99	92
query43	599	544	528	528
query44	1085	718	757	718
query45	279	252	258	252
query46	1062	773	739	739
query47	1936	1890	1899	1890
query48	369	289	301	289
query49	752	395	393	393
query50	768	391	392	391
query51	6776	6669	6727	6669
query52	103	91	90	90
query53	345	280	293	280
query54	271	245	238	238
query55	81	80	72	72
query56	242	226	229	226
query57	1221	1115	1139	1115
query58	214	212	195	195
query59	3453	3457	3187	3187
query60	249	243	227	227
query61	105	86	86	86
query62	561	469	431	431
query63	305	277	272	272
query64	8169	7144	7092	7092
query65	3103	3042	3074	3042
query66	796	338	332	332
query67	15362	14974	15003	14974
query68	5194	538	551	538
query69	484	294	315	294
query70	1152	1126	1104	1104
query71	391	263	270	263
query72	7123	2608	2427	2427
query73	715	329	336	329
query74	6534	6082	6072	6072
query75	3340	2682	2709	2682
query76	2796	1019	1040	1019
query77	414	259	263	259
query78	10786	10364	10334	10334
query79	2753	530	523	523
query80	1971	429	425	425
query81	524	224	225	224
query82	758	102	93	93
query83	274	169	166	166
query84	269	88	84	84
query85	1785	268	275	268
query86	517	323	298	298
query87	3276	3144	3133	3133
query88	4104	2431	2464	2431
query89	475	389	370	370
query90	2044	182	184	182
query91	124	98	99	98
query92	66	49	50	49
query93	3889	518	507	507
query94	1262	182	181	181
query95	388	303	310	303
query96	598	270	263	263
query97	3112	2966	2907	2907
query98	235	222	218	218
query99	1175	833	866	833
Total cold run time: 274775 ms
Total hot run time: 186820 ms

dataroaring

LGTM

github-actions · 2024-04-29T02:53:30Z

PR approved by at least one committer and no changes requested.

…ache#34220)

…4220) (#38260) Pick #34220

morningman previously approved these changes Apr 28, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 28, 2024

github-actions bot added the reviewed label Apr 28, 2024

dataroaring previously approved these changes Apr 28, 2024

View reviewed changes

morrySnow previously approved these changes Apr 28, 2024

View reviewed changes

[fix](insert) Insert init plan error may forget to abrot txn

05f3cb7

mymeiyi dismissed stale reviews from morrySnow, dataroaring, and morningman via 05f3cb7 April 28, 2024 07:20

mymeiyi force-pushed the fix-insert branch from 955cc76 to 05f3cb7 Compare April 28, 2024 07:20

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 28, 2024

mymeiyi changed the title ~~[fix](insert) Insert init plan error may forget to abrot txn~~ [fix](insert) Insert init plan error may forget to abort txn Apr 28, 2024

gavinchou approved these changes Apr 28, 2024

View reviewed changes

dataroaring approved these changes Apr 29, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 29, 2024

dataroaring merged commit 4ab7ac0 into apache:master Apr 29, 2024

mymeiyi added a commit to mymeiyi/doris that referenced this pull request Jul 23, 2024

[fix](insert) Pick Insert init plan error may forget to abrot txn (ap…

07a6bc2

…ache#34220)

mymeiyi mentioned this pull request Jul 23, 2024

[fix](insert) Pick Insert init plan error may forget to abort txn (#34220) #38260

Merged

dataroaring pushed a commit that referenced this pull request Jul 23, 2024

[fix](insert) Pick Insert init plan error may forget to abort txn (#3…

b591837

…4220) (#38260) Pick #34220

dataroaring added the dev/2.1.6-merged label Jul 23, 2024

dataroaring added dev/3.0.0-merged dev/2.1.6-merged and removed dev/2.1.6-merged labels Jul 31, 2024

[fix](insert) Insert init plan error may forget to abort txn #34220

[fix](insert) Insert init plan error may forget to abort txn #34220

Uh oh!

Conversation

mymeiyi commented Apr 28, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented Apr 28, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

mymeiyi commented Apr 28, 2024

Uh oh!

github-actions bot commented Apr 28, 2024

Uh oh!

github-actions bot commented Apr 28, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Apr 28, 2024

Uh oh!

doris-robot commented Apr 28, 2024

Uh oh!

mymeiyi commented Apr 28, 2024

Uh oh!

doris-robot commented Apr 28, 2024

Uh oh!

doris-robot commented Apr 28, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants