[branch-2.0](cherry-pick) mishandling of exceptions in the publish phase may result in data loss (#30366)

[zhannngchen]

Proposed changes

Issue Number: close #xxx

cherry-pick #30366

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

[doris-robot]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

[zhannngchen]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[doris-robot]

TeamCity be ut coverage result:
Function Coverage: 37.80% (8055/21308)
Line Coverage: 29.48% (65787/223166)
Region Coverage: 28.93% (33840/116958)
Branch Coverage: 24.79% (17375/70084)
Coverage Report: http://coverage.selectdb-in.cc/coverage/65d1a4f6de9b51298b251c7aff499a996f6d5810_65d1a4f6de9b51298b251c7aff499a996f6d5810/report/index.html

[doris-robot]

TPC-H: Total hot run time: 49823 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 65d1a4f6de9b51298b251c7aff499a996f6d5810, data reload: false

------ Round 1 ----------------------------------
q1	17635	4399	4382	4382
q2	2056	153	140	140
q3	10449	1852	1968	1852
q4	10321	1207	1315	1207
q5	8496	3973	3989	3973
q6	235	124	124	124
q7	2056	1587	1591	1587
q8	9288	2710	2746	2710
q9	11166	10811	10538	10538
q10	8610	3503	3486	3486
q11	410	236	257	236
q12	467	298	303	298
q13	18334	3966	4032	3966
q14	349	317	306	306
q15	497	458	465	458
q16	702	579	588	579
q17	1117	973	950	950
q18	7372	6741	6873	6741
q19	1688	1549	1508	1508
q20	524	307	299	299
q21	4461	4113	4090	4090
q22	507	393	402	393
Total cold run time: 116740 ms
Total hot run time: 49823 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4359	4300	4328	4300
q2	325	218	221	218
q3	4151	4118	4126	4118
q4	2762	2727	2742	2727
q5	7248	7245	7173	7173
q6	236	120	120	120
q7	3221	2744	2798	2744
q8	4321	4446	4420	4420
q9	17669	17244	17337	17244
q10	4228	4301	4221	4221
q11	771	659	673	659
q12	1024	840	849	840
q13	6730	3720	3720	3720
q14	450	420	429	420
q15	498	460	459	459
q16	763	708	705	705
q17	3759	3785	3837	3785
q18	8813	8717	8745	8717
q19	1735	1702	1652	1652
q20	2390	2129	2125	2125
q21	8493	8476	8411	8411
q22	1028	945	960	945
Total cold run time: 84974 ms
Total hot run time: 79723 ms

[doris-robot]

TPC-DS: Total hot run time: 200772 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 65d1a4f6de9b51298b251c7aff499a996f6d5810, data reload: false

query1	925	392	376	376
query2	6543	2192	2081	2081
query3	6927	202	198	198
query4	19878	17902	17998	17902
query5	19721	6524	6488	6488
query6	279	212	207	207
query7	4146	300	290	290
query8	294	225	235	225
query9	3126	2687	2623	2623
query10	404	286	295	286
query11	11401	10784	10803	10784
query12	123	78	73	73
query13	5631	670	641	641
query14	17684	13627	13360	13360
query15	365	233	237	233
query16	6477	276	264	264
query17	1706	1473	884	884
query18	2303	415	400	400
query19	206	144	141	141
query20	74	75	73	73
query21	185	94	89	89
query22	5203	4998	5157	4998
query23	32415	31947	31715	31715
query24	7062	6495	6540	6495
query25	533	451	416	416
query26	522	159	156	156
query27	1894	291	290	290
query28	6152	2290	2246	2246
query29	2904	2644	2576	2576
query30	239	161	161	161
query31	922	716	710	710
query32	66	58	58	58
query33	399	242	253	242
query34	851	471	471	471
query35	1150	857	890	857
query36	1597	1258	1159	1159
query37	89	61	61	61
query38	3096	2870	2924	2870
query39	1358	1324	1296	1296
query40	200	93	101	93
query41	36	33	32	32
query42	88	83	83	83
query43	692	553	584	553
query44	1117	722	718	718
query45	240	221	224	221
query46	1228	966	986	966
query47	1833	1941	1722	1722
query48	974	667	670	667
query49	616	364	369	364
query50	838	598	636	598
query51	4867	4618	4674	4618
query52	82	70	69	69
query53	440	314	313	313
query54	2639	2454	2444	2444
query55	95	83	76	76
query56	212	201	198	198
query57	1172	1104	1158	1104
query58	205	199	199	199
query59	3545	3433	3091	3091
query60	204	197	220	197
query61	89	87	86	86
query62	836	510	501	501
query63	487	332	337	332
query64	2574	1310	1265	1265
query65	3606	3529	3537	3529
query66	765	372	366	366
query67	16096	15367	15155	15155
query68	8771	648	655	648
query69	568	338	340	338
query70	1599	1344	1400	1344
query71	423	296	323	296
query72	6488	3452	3508	3452
query73	735	319	328	319
query74	6306	5890	5917	5890
query75	5009	3770	3658	3658
query76	5097	1145	1170	1145
query77	705	248	256	248
query78	12753	11835	11895	11835
query79	6777	629	652	629
query80	1324	398	383	383
query81	487	231	223	223
query82	1502	97	99	97
query83	169	136	139	136
query84	257	68	68	68
query85	1057	294	283	283
query86	340	294	298	294
query87	3244	3002	2930	2930
query88	4943	2366	2368	2366
query89	336	304	281	281
query90	1976	213	210	210
query91	150	127	118	118
query92	56	50	52	50
query93	2464	579	591	579
query94	804	204	200	200
query95	1114	1059	1034	1034
query96	650	333	328	328
query97	6518	6377	6415	6377
query98	178	176	167	167
query99	2903	891	899	891
Total cold run time: 306329 ms
Total hot run time: 200772 ms

[doris-robot]

ClickBench: Total hot run time: 30.6 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 65d1a4f6de9b51298b251c7aff499a996f6d5810, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.24	0.05	0.05
query4	1.81	0.07	0.07
query5	0.53	0.52	0.52
query6	1.22	0.66	0.61
query7	0.02	0.02	0.02
query8	0.04	0.02	0.02
query9	0.52	0.47	0.48
query10	0.53	0.53	0.54
query11	0.11	0.09	0.08
query12	0.12	0.09	0.10
query13	0.62	0.61	0.60
query14	0.78	0.78	0.77
query15	0.79	0.75	0.77
query16	0.36	0.36	0.36
query17	1.02	1.01	1.02
query18	0.23	0.25	0.24
query19	1.94	1.80	1.84
query20	0.01	0.01	0.01
query21	15.48	0.57	0.56
query22	2.04	3.15	1.44
query23	17.15	0.98	0.95
query24	4.45	0.98	1.44
query25	0.36	0.12	0.04
query26	0.52	0.17	0.15
query27	0.04	0.05	0.04
query28	8.25	0.74	0.72
query29	12.67	2.32	2.29
query30	0.62	0.59	0.52
query31	2.82	0.39	0.37
query32	3.40	0.49	0.50
query33	3.08	3.08	3.11
query34	15.25	4.83	4.80
query35	4.87	4.87	4.89
query36	1.04	1.02	1.02
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.15	0.14
query41	0.06	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.01
Total cold run time: 103.39 s
Total hot run time: 30.6 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 65d1a4f6de9b51298b251c7aff499a996f6d5810 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       20.7 seconds inserted 10000000 Rows, about 483K ops/s