(cloud-merge) Support to abort txn when coordinate be restart and do schema change #37669

Lchangliang · 2024-07-11T09:33:15Z

When do schema change, it will check whether transactions of the given database which txnId is less than 'watershedTxnId' are finished. If is not, schema change will be waiting.
When the coordinator restart (whatever fe/be), the txn belong the cooridinator will be hang until timeout. So we need to abort it in time.
There are two optimizations.

Check BE's lastStartTime when get it heart beat. If current lastStartTime is large than lastStartTime in fe memory. Abort all the hang txns belong this BE.
Check conflict txns when do schema change. If txns is failed (Maybe coordinator be/fe restart), abort it directly.

doris-robot · 2024-07-11T09:33:21Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Lchangliang · 2024-07-11T09:41:39Z

run buildall

Lchangliang · 2024-07-15T06:46:15Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-07-15T06:51:05Z

cloud/src/meta-service/meta_service_txn.cpp

 // specific language governing permissions and limitations
 // under the License.

+#include <gen_cpp/cloud.pb.h>


warning: 'gen_cpp/cloud.pb.h' file not found [clang-diagnostic-error]

#include <gen_cpp/cloud.pb.h> ^

Lchangliang · 2024-07-15T08:12:00Z

run buildall

Lchangliang · 2024-07-15T08:31:12Z

run buildall

doris-robot · 2024-07-15T08:53:26Z

TPC-H: Total hot run time: 39787 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5db13f08bb5b1cb095c64e41202d0d82d1c85242, data reload: false

------ Round 1 ----------------------------------
q1	17689	4792	4369	4369
q2	2309	193	191	191
q3	10531	1178	1136	1136
q4	10238	711	748	711
q5	7543	2699	2672	2672
q6	222	136	137	136
q7	952	605	594	594
q8	9230	2044	2056	2044
q9	8731	6564	6486	6486
q10	8809	3749	3810	3749
q11	451	234	239	234
q12	386	226	219	219
q13	17762	2988	2948	2948
q14	275	248	238	238
q15	528	481	486	481
q16	497	384	379	379
q17	972	665	670	665
q18	8073	7486	7436	7436
q19	6785	1315	1256	1256
q20	671	317	333	317
q21	5139	3247	3302	3247
q22	345	279	283	279
Total cold run time: 118138 ms
Total hot run time: 39787 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4373	4241	4290	4241
q2	367	276	253	253
q3	3009	2919	2935	2919
q4	2037	1657	1759	1657
q5	5664	5546	5468	5468
q6	219	130	130	130
q7	2209	1854	1844	1844
q8	3271	3406	3417	3406
q9	8722	8785	8841	8785
q10	4064	3899	3728	3728
q11	579	493	508	493
q12	807	640	641	640
q13	15997	3187	3153	3153
q14	321	284	277	277
q15	520	479	498	479
q16	495	447	448	447
q17	1832	1533	1490	1490
q18	8012	8007	7895	7895
q19	1781	1541	1513	1513
q20	2839	1888	1883	1883
q21	5139	4991	4664	4664
q22	587	494	496	494
Total cold run time: 72844 ms
Total hot run time: 55859 ms

doris-robot · 2024-07-15T09:04:32Z

TPC-DS: Total hot run time: 173334 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5db13f08bb5b1cb095c64e41202d0d82d1c85242, data reload: false

query1	922	380	357	357
query2	6447	1862	1835	1835
query3	6646	207	219	207
query4	23990	17425	17295	17295
query5	3707	469	470	469
query6	258	168	169	168
query7	4591	285	281	281
query8	247	199	193	193
query9	8507	2350	2340	2340
query10	434	291	274	274
query11	11805	10073	10021	10021
query12	115	109	82	82
query13	1646	361	360	360
query14	10138	7257	6961	6961
query15	210	166	162	162
query16	7414	322	309	309
query17	1367	535	522	522
query18	1954	275	270	270
query19	195	155	145	145
query20	91	80	80	80
query21	209	126	124	124
query22	4620	4200	3919	3919
query23	34424	33746	33860	33746
query24	11068	2925	2894	2894
query25	600	389	384	384
query26	706	153	147	147
query27	2315	273	271	271
query28	6078	2022	2016	2016
query29	892	640	635	635
query30	261	158	156	156
query31	968	770	771	770
query32	95	53	54	53
query33	693	300	289	289
query34	935	494	486	486
query35	715	582	564	564
query36	1144	976	966	966
query37	139	82	87	82
query38	2949	2905	2827	2827
query39	897	862	824	824
query40	198	119	114	114
query41	46	42	41	41
query42	120	97	102	97
query43	525	463	464	463
query44	1151	717	739	717
query45	194	162	166	162
query46	1084	751	747	747
query47	1837	1775	1794	1775
query48	358	298	284	284
query49	843	404	422	404
query50	785	398	388	388
query51	6859	6773	6790	6773
query52	102	93	95	93
query53	349	280	280	280
query54	921	443	439	439
query55	74	72	74	72
query56	297	320	265	265
query57	1120	1031	1087	1031
query58	266	247	254	247
query59	2845	2738	2576	2576
query60	298	268	278	268
query61	96	94	92	92
query62	777	645	650	645
query63	323	281	289	281
query64	9132	2185	7430	2185
query65	3155	3108	3098	3098
query66	726	334	330	330
query67	15764	15307	15110	15110
query68	4479	515	517	515
query69	585	532	336	336
query70	1206	1129	1141	1129
query71	378	282	274	274
query72	6844	5454	5829	5454
query73	743	319	328	319
query74	6109	5730	5670	5670
query75	3345	2730	2648	2648
query76	2389	1007	934	934
query77	622	307	288	288
query78	9614	9182	8943	8943
query79	2567	529	539	529
query80	1717	494	479	479
query81	611	222	219	219
query82	771	144	135	135
query83	289	175	164	164
query84	262	90	89	89
query85	1067	323	369	323
query86	402	315	288	288
query87	3319	3112	3132	3112
query88	3855	2477	2444	2444
query89	471	387	377	377
query90	1596	195	203	195
query91	128	102	102	102
query92	60	51	51	51
query93	2216	500	499	499
query94	786	215	214	214
query95	406	322	324	322
query96	616	283	278	278
query97	3214	3085	3090	3085
query98	215	203	190	190
query99	1641	1278	1268	1268
Total cold run time: 273160 ms
Total hot run time: 173334 ms

doris-robot · 2024-07-15T09:10:06Z

ClickBench: Total hot run time: 30.29 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5db13f08bb5b1cb095c64e41202d0d82d1c85242, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.04
query4	1.67	0.08	0.08
query5	0.51	0.49	0.49
query6	1.15	0.73	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.55	0.54	0.55
query11	0.14	0.11	0.12
query12	0.15	0.12	0.12
query13	0.59	0.59	0.58
query14	0.77	0.78	0.76
query15	0.86	0.82	0.82
query16	0.36	0.37	0.37
query17	1.00	0.96	1.06
query18	0.23	0.21	0.21
query19	1.84	1.72	1.71
query20	0.02	0.01	0.01
query21	15.39	0.75	0.65
query22	4.59	7.93	1.58
query23	18.29	1.33	1.30
query24	2.14	0.22	0.22
query25	0.14	0.08	0.09
query26	0.30	0.20	0.20
query27	0.45	0.23	0.22
query28	13.23	1.03	1.00
query29	12.60	3.30	3.33
query30	0.25	0.08	0.05
query31	2.87	0.39	0.38
query32	3.26	0.48	0.46
query33	2.91	2.96	2.94
query34	16.94	4.43	4.34
query35	4.39	4.43	4.38
query36	0.64	0.46	0.48
query37	0.18	0.17	0.15
query38	0.15	0.14	0.14
query39	0.05	0.03	0.05
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.92 s
Total hot run time: 30.29 s

Lchangliang · 2024-07-16T16:32:10Z

run buildall

doris-robot · 2024-07-16T17:53:56Z

TPC-H: Total hot run time: 39994 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 70332d6f54e506131bafc2390ad62425fc6e280e, data reload: false

------ Round 1 ----------------------------------
q1	18150	4688	4306	4306
q2	2029	203	194	194
q3	10541	1173	1047	1047
q4	10211	869	856	856
q5	7592	2768	2614	2614
q6	223	141	139	139
q7	956	596	628	596
q8	9233	2105	2093	2093
q9	8797	6572	6526	6526
q10	8750	3811	3774	3774
q11	456	238	241	238
q12	399	230	225	225
q13	17768	3014	3030	3014
q14	289	233	237	233
q15	527	484	482	482
q16	499	386	376	376
q17	985	660	701	660
q18	8044	7501	7468	7468
q19	5681	1490	1371	1371
q20	649	323	324	323
q21	4851	3174	3212	3174
q22	353	287	285	285
Total cold run time: 116983 ms
Total hot run time: 39994 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4419	4253	4233	4233
q2	375	295	261	261
q3	3035	2908	2874	2874
q4	2016	1640	1703	1640
q5	5665	5551	5771	5551
q6	227	135	133	133
q7	2197	1869	1811	1811
q8	3283	3466	3431	3431
q9	8759	8822	8886	8822
q10	4086	3909	3779	3779
q11	602	514	504	504
q12	823	632	625	625
q13	17144	3179	3203	3179
q14	315	288	283	283
q15	531	481	505	481
q16	485	450	430	430
q17	1810	1519	1520	1519
q18	8039	7851	7781	7781
q19	1703	1583	1595	1583
q20	2325	1892	1847	1847
q21	5093	4947	4822	4822
q22	584	500	499	499
Total cold run time: 73516 ms
Total hot run time: 56088 ms

doris-robot · 2024-07-16T18:05:07Z

TPC-DS: Total hot run time: 172919 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 70332d6f54e506131bafc2390ad62425fc6e280e, data reload: false

query1	911	381	369	369
query2	6464	1921	1814	1814
query3	6630	210	221	210
query4	28189	17642	17331	17331
query5	3633	514	494	494
query6	280	179	167	167
query7	4585	292	299	292
query8	238	187	191	187
query9	8647	2390	2377	2377
query10	439	292	280	280
query11	11173	10026	9959	9959
query12	114	84	80	80
query13	1642	366	353	353
query14	10734	7536	7804	7536
query15	220	167	167	167
query16	7629	327	314	314
query17	1402	573	540	540
query18	1828	289	301	289
query19	194	150	147	147
query20	88	82	81	81
query21	208	126	125	125
query22	4383	3998	3912	3912
query23	33847	33657	33482	33482
query24	11205	2927	2873	2873
query25	630	392	377	377
query26	1181	157	158	157
query27	2354	283	288	283
query28	6890	2070	2085	2070
query29	949	660	636	636
query30	258	159	160	159
query31	1010	752	772	752
query32	102	53	57	53
query33	780	295	296	295
query34	1025	491	504	491
query35	683	585	588	585
query36	1119	953	1007	953
query37	148	88	92	88
query38	2965	2829	2855	2829
query39	905	869	851	851
query40	211	128	128	128
query41	47	45	45	45
query42	116	106	104	104
query43	500	472	465	465
query44	1290	738	725	725
query45	195	158	160	158
query46	1085	736	729	729
query47	1833	1761	1761	1761
query48	356	301	288	288
query49	846	465	418	418
query50	777	384	391	384
query51	6864	6821	6776	6776
query52	102	88	94	88
query53	364	294	278	278
query54	881	452	444	444
query55	74	76	74	74
query56	290	272	264	264
query57	1139	1034	1031	1031
query58	255	245	246	245
query59	2769	2668	2581	2581
query60	306	275	287	275
query61	103	95	98	95
query62	814	630	663	630
query63	328	295	291	291
query64	9439	2260	1671	1671
query65	3163	3085	3109	3085
query66	727	332	334	332
query67	15259	14882	14798	14798
query68	4523	542	542	542
query69	657	450	349	349
query70	1095	1187	1094	1094
query71	457	285	281	281
query72	7240	5894	5411	5411
query73	758	337	328	328
query74	6228	5658	5654	5654
query75	3415	2724	2730	2724
query76	3174	897	982	897
query77	619	306	315	306
query78	9690	8961	8960	8960
query79	2883	524	548	524
query80	2716	485	486	485
query81	592	227	226	226
query82	737	132	135	132
query83	316	174	178	174
query84	274	89	87	87
query85	2034	323	368	323
query86	482	333	302	302
query87	3294	3097	3143	3097
query88	3878	2374	2378	2374
query89	490	389	395	389
query90	1878	194	192	192
query91	131	98	104	98
query92	59	49	47	47
query93	4464	528	514	514
query94	1272	213	209	209
query95	408	317	316	316
query96	626	273	273	273
query97	3210	3058	3026	3026
query98	222	206	194	194
query99	1562	1296	1270	1270
Total cold run time: 285199 ms
Total hot run time: 172919 ms

doris-robot · 2024-07-16T18:10:41Z

ClickBench: Total hot run time: 30.32 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 70332d6f54e506131bafc2390ad62425fc6e280e, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.03
query3	0.22	0.05	0.05
query4	1.67	0.07	0.08
query5	0.49	0.47	0.48
query6	1.14	0.71	0.73
query7	0.03	0.02	0.01
query8	0.06	0.05	0.05
query9	0.54	0.47	0.48
query10	0.53	0.54	0.54
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.59	0.58	0.58
query14	0.76	0.79	0.78
query15	0.87	0.81	0.82
query16	0.36	0.36	0.35
query17	1.04	0.95	0.95
query18	0.23	0.22	0.22
query19	1.77	1.70	1.72
query20	0.02	0.01	0.01
query21	15.42	0.74	0.65
query22	4.59	7.45	1.51
query23	18.23	1.38	1.30
query24	2.08	0.23	0.22
query25	0.15	0.09	0.08
query26	0.30	0.21	0.21
query27	0.46	0.23	0.22
query28	13.28	1.04	1.00
query29	12.61	3.45	3.37
query30	0.25	0.06	0.06
query31	2.89	0.39	0.39
query32	3.27	0.48	0.48
query33	2.94	2.94	2.93
query34	17.08	4.36	4.38
query35	4.40	4.41	4.42
query36	0.66	0.50	0.47
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.16	0.13	0.13
query41	0.09	0.04	0.05
query42	0.05	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 110.08 s
Total hot run time: 30.32 s

Lchangliang · 2024-07-17T02:05:27Z

run cloud_p0

cloud/src/meta-service/meta_service_txn.cpp

fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java

cloud/src/meta-service/meta_service.h

gensrc/proto/cloud.proto

fe/fe-core/src/main/java/org/apache/doris/transaction/GlobalTransactionMgr.java

Lchangliang · 2024-07-22T09:27:28Z

run buildall

Lchangliang · 2024-07-22T09:28:54Z

run buildall

Lchangliang · 2024-07-22T09:36:23Z

run buildall

doris-robot · 2024-07-22T10:37:56Z

TPC-H: Total hot run time: 40310 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b2bfd9c7e4bd076f1608ed121c1effb4ca2ea144, data reload: false

------ Round 1 ----------------------------------
q1	18792	4631	4415	4415
q2	2482	202	193	193
q3	11578	1210	1028	1028
q4	10618	908	792	792
q5	7959	2815	2765	2765
q6	222	137	142	137
q7	976	613	609	609
q8	9473	2150	2122	2122
q9	8896	6621	6610	6610
q10	8766	3822	3805	3805
q11	464	239	240	239
q12	401	230	229	229
q13	17768	2982	3019	2982
q14	284	232	235	232
q15	538	483	479	479
q16	488	383	382	382
q17	987	678	704	678
q18	8127	7517	7456	7456
q19	4935	1410	1317	1317
q20	679	328	338	328
q21	4966	3276	3228	3228
q22	346	284	292	284
Total cold run time: 119745 ms
Total hot run time: 40310 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4463	4258	4280	4258
q2	387	267	263	263
q3	3055	2808	2759	2759
q4	1880	1642	1610	1610
q5	5300	5339	5350	5339
q6	215	130	131	130
q7	2134	1775	1699	1699
q8	3224	3420	3320	3320
q9	8553	8521	8510	8510
q10	3928	3777	3756	3756
q11	583	478	510	478
q12	803	642	603	603
q13	17498	2963	2985	2963
q14	295	268	267	267
q15	518	474	481	474
q16	475	408	419	408
q17	1797	1492	1461	1461
q18	7673	7679	7346	7346
q19	1724	1352	1571	1352
q20	2004	1823	1808	1808
q21	4845	4705	4796	4705
q22	606	515	509	509
Total cold run time: 71960 ms
Total hot run time: 54018 ms

doris-robot · 2024-07-22T10:49:32Z

TPC-DS: Total hot run time: 173016 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b2bfd9c7e4bd076f1608ed121c1effb4ca2ea144, data reload: false

query1	903	372	366	366
query2	6444	1916	1820	1820
query3	6667	211	215	211
query4	28187	17587	17127	17127
query5	4305	478	472	472
query6	270	164	165	164
query7	4607	290	284	284
query8	254	208	198	198
query9	8629	2498	2457	2457
query10	442	295	278	278
query11	12720	10155	9961	9961
query12	129	83	84	83
query13	1636	371	370	370
query14	10271	7724	7670	7670
query15	232	167	169	167
query16	7845	478	468	468
query17	1594	560	550	550
query18	1964	281	282	281
query19	204	151	153	151
query20	88	84	84	84
query21	206	126	134	126
query22	4427	4002	3928	3928
query23	33989	33314	33809	33314
query24	12112	2913	2871	2871
query25	680	384	388	384
query26	1811	151	150	150
query27	2972	276	271	271
query28	7291	2024	2023	2023
query29	1142	651	626	626
query30	293	152	164	152
query31	964	748	750	748
query32	105	55	56	55
query33	781	357	348	348
query34	882	483	483	483
query35	861	754	748	748
query36	1096	939	925	925
query37	279	85	82	82
query38	2851	2800	2767	2767
query39	851	818	856	818
query40	276	124	121	121
query41	52	48	46	46
query42	119	102	103	102
query43	497	489	464	464
query44	1187	719	711	711
query45	189	166	162	162
query46	1099	721	723	721
query47	1887	1796	1797	1796
query48	372	296	288	288
query49	1166	428	423	423
query50	782	404	398	398
query51	6812	6748	6514	6514
query52	106	97	95	95
query53	363	293	289	289
query54	867	465	458	458
query55	74	75	74	74
query56	300	280	410	280
query57	1150	1059	1063	1059
query58	249	238	264	238
query59	2751	2537	2652	2537
query60	299	269	283	269
query61	109	94	94	94
query62	805	638	629	629
query63	323	299	293	293
query64	10421	2267	1679	1679
query65	3135	3113	3147	3113
query66	1188	327	330	327
query67	15691	15043	14899	14899
query68	7810	537	533	533
query69	742	467	362	362
query70	1221	1093	1055	1055
query71	522	274	290	274
query72	9118	5523	5707	5523
query73	802	325	325	325
query74	6059	5717	5647	5647
query75	4771	2707	2714	2707
query76	4860	932	885	885
query77	795	304	299	299
query78	9748	9023	9031	9023
query79	8272	521	516	516
query80	2371	479	488	479
query81	592	227	223	223
query82	777	127	128	127
query83	271	162	169	162
query84	281	86	88	86
query85	1468	351	295	295
query86	443	324	305	305
query87	3326	3100	3109	3100
query88	4676	2458	2526	2458
query89	509	375	377	375
query90	2145	195	192	192
query91	127	100	98	98
query92	58	51	49	49
query93	6676	510	496	496
query94	1473	274	302	274
query95	418	316	332	316
query96	623	275	268	268
query97	3180	3149	3053	3053
query98	216	198	198	198
query99	1695	1316	1253	1253
Total cold run time: 309298 ms
Total hot run time: 173016 ms

doris-robot · 2024-07-22T10:55:03Z

ClickBench: Total hot run time: 30.66 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b2bfd9c7e4bd076f1608ed121c1effb4ca2ea144, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.08
query5	0.50	0.48	0.48
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.55	0.48	0.48
query10	0.55	0.54	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.61	0.59	0.58
query14	0.76	0.77	0.79
query15	0.84	0.81	0.81
query16	0.35	0.35	0.35
query17	0.96	1.01	1.04
query18	0.23	0.22	0.22
query19	1.90	1.68	1.66
query20	0.01	0.01	0.01
query21	15.39	0.78	0.68
query22	4.55	6.61	1.97
query23	18.26	1.41	1.28
query24	2.10	0.26	0.22
query25	0.14	0.09	0.08
query26	0.30	0.21	0.21
query27	0.45	0.24	0.23
query28	13.24	1.02	1.00
query29	12.62	3.24	3.24
query30	0.25	0.06	0.06
query31	2.88	0.40	0.38
query32	3.27	0.46	0.46
query33	2.90	2.93	2.90
query34	17.08	4.41	4.38
query35	4.40	4.43	4.49
query36	0.63	0.48	0.45
query37	0.20	0.16	0.16
query38	0.17	0.15	0.15
query39	0.05	0.03	0.03
query40	0.16	0.12	0.13
query41	0.09	0.05	0.04
query42	0.06	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.02 s
Total hot run time: 30.66 s

Lchangliang · 2024-07-25T03:55:38Z

run external

cloud/src/meta-service/meta_service_txn.cpp

Lchangliang · 2024-07-29T06:19:26Z

run buildall

fe/fe-core/src/main/java/org/apache/doris/transaction/GlobalTransactionMgr.java

cloud/src/meta-service/meta_service_txn.cpp

Lchangliang · 2024-07-29T07:31:29Z

run buildall

doris-robot · 2024-07-29T08:23:11Z

TPC-H: Total hot run time: 41733 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 127cd9ef441ec227b80a42b7fec29ab7958f6a01, data reload: false

------ Round 1 ----------------------------------
q1	17624	4469	4038	4038
q2	2025	214	213	213
q3	10579	1329	1384	1329
q4	10323	838	984	838
q5	7663	2993	2983	2983
q6	221	138	138	138
q7	1048	628	636	628
q8	9431	1780	1947	1780
q9	8503	6615	6627	6615
q10	8713	3847	3844	3844
q11	436	252	247	247
q12	408	231	233	231
q13	17792	2970	2945	2945
q14	267	248	242	242
q15	532	490	491	490
q16	534	413	393	393
q17	990	902	876	876
q18	8004	7419	7324	7324
q19	1388	1214	1217	1214
q20	557	330	338	330
q21	5352	4750	4830	4750
q22	362	294	285	285
Total cold run time: 112752 ms
Total hot run time: 41733 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4175	4012	4006	4006
q2	329	230	225	225
q3	2985	3025	3156	3025
q4	2009	2053	1983	1983
q5	5638	5536	5488	5488
q6	217	142	142	142
q7	2136	1748	1877	1748
q8	3362	3359	3366	3359
q9	8727	8655	8820	8655
q10	3959	4032	4015	4015
q11	554	458	452	452
q12	739	580	636	580
q13	16437	3112	3198	3112
q14	311	275	284	275
q15	535	483	500	483
q16	489	445	445	445
q17	1775	1762	1728	1728
q18	8190	7692	7653	7653
q19	3170	1712	1716	1712
q20	2051	1865	1856	1856
q21	5856	5596	5436	5436
q22	524	438	469	438
Total cold run time: 74168 ms
Total hot run time: 56816 ms

doris-robot · 2024-07-29T08:34:11Z

TPC-DS: Total hot run time: 171104 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 127cd9ef441ec227b80a42b7fec29ab7958f6a01, data reload: false

query1	914	384	369	369
query2	6480	1768	1721	1721
query3	6653	223	226	223
query4	19508	17284	17153	17153
query5	3648	521	552	521
query6	273	188	173	173
query7	4606	319	309	309
query8	259	205	212	205
query9	8525	2448	2429	2429
query10	432	287	276	276
query11	10468	10023	10011	10011
query12	120	87	95	87
query13	1625	381	382	381
query14	9511	8512	7434	7434
query15	201	164	166	164
query16	6846	405	438	405
query17	940	577	542	542
query18	1934	290	296	290
query19	193	145	145	145
query20	89	88	89	88
query21	210	103	101	101
query22	4157	3875	3925	3875
query23	34026	33710	33595	33595
query24	10488	3118	3113	3113
query25	721	429	415	415
query26	1723	165	158	158
query27	3157	306	298	298
query28	7561	2066	2036	2036
query29	1289	467	439	439
query30	250	156	152	152
query31	943	767	807	767
query32	101	60	56	56
query33	689	347	340	340
query34	931	539	524	524
query35	894	772	739	739
query36	1043	899	879	879
query37	305	86	85	85
query38	2886	2812	2802	2802
query39	855	808	816	808
query40	266	113	115	113
query41	48	47	44	44
query42	117	99	97	97
query43	487	420	432	420
query44	1211	727	736	727
query45	207	177	174	174
query46	1095	828	815	815
query47	1798	1708	1694	1694
query48	371	294	289	289
query49	940	445	425	425
query50	933	452	446	446
query51	6871	6693	6713	6693
query52	104	94	92	92
query53	265	182	179	179
query54	624	468	472	468
query55	80	77	76	76
query56	288	267	265	265
query57	1127	1052	1045	1045
query58	277	279	279	279
query59	2753	2422	2457	2422
query60	314	302	287	287
query61	122	115	118	115
query62	895	672	671	671
query63	221	189	193	189
query64	5750	2034	2003	2003
query65	3392	3167	3145	3145
query66	1340	354	352	352
query67	15743	14908	14597	14597
query68	6555	599	612	599
query69	686	390	332	332
query70	1150	1071	1075	1071
query71	514	287	284	284
query72	7739	2881	2683	2683
query73	996	341	336	336
query74	6018	5661	5620	5620
query75	3785	2752	2768	2752
query76	5309	1405	1430	1405
query77	786	325	310	310
query78	9618	9001	8978	8978
query79	4301	561	554	554
query80	2171	535	521	521
query81	555	228	225	225
query82	775	136	133	133
query83	244	176	173	173
query84	276	84	85	84
query85	1536	316	319	316
query86	462	294	277	277
query87	3272	3063	3088	3063
query88	4200	2477	2510	2477
query89	471	293	298	293
query90	1855	197	195	195
query91	129	105	101	101
query92	63	54	51	51
query93	5165	644	647	644
query94	814	293	302	293
query95	385	279	283	279
query96	616	286	278	278
query97	3224	3017	3065	3017
query98	225	208	209	208
query99	1683	1322	1330	1322
Total cold run time: 278671 ms
Total hot run time: 171104 ms

doris-robot · 2024-07-29T08:39:43Z

ClickBench: Total hot run time: 29.87 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 127cd9ef441ec227b80a42b7fec29ab7958f6a01, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.69	0.08	0.07
query5	0.50	0.49	0.49
query6	1.13	0.72	0.71
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.56	0.52	0.53
query10	0.58	0.56	0.58
query11	0.15	0.11	0.11
query12	0.16	0.12	0.13
query13	0.62	0.60	0.60
query14	0.78	0.80	0.78
query15	0.89	0.87	0.86
query16	0.35	0.35	0.36
query17	1.00	0.99	0.97
query18	0.22	0.21	0.22
query19	1.81	1.71	1.82
query20	0.01	0.01	0.01
query21	15.43	0.75	0.64
query22	4.09	7.63	1.12
query23	17.96	1.31	1.34
query24	2.27	0.22	0.22
query25	0.19	0.08	0.08
query26	0.31	0.21	0.22
query27	0.46	0.24	0.24
query28	13.17	1.00	0.97
query29	12.59	3.29	3.30
query30	0.25	0.06	0.05
query31	2.85	0.40	0.41
query32	3.25	0.50	0.48
query33	2.92	2.98	2.97
query34	15.44	4.28	4.29
query35	4.32	4.31	4.29
query36	0.69	0.49	0.50
query37	0.20	0.16	0.17
query38	0.18	0.15	0.15
query39	0.05	0.03	0.04
query40	0.16	0.13	0.13
query41	0.10	0.04	0.05
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 107.84 s
Total hot run time: 29.87 s

github-actions · 2024-07-30T02:26:02Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-30T02:26:04Z

PR approved by anyone and no changes requested.

…schema change (#37669) When do schema change, it will check whether transactions of the given database which txnId is less than 'watershedTxnId' are finished. If is not, schema change will be waiting. When the coordinator restart (whatever fe/be), the txn belong the cooridinator will be hang until timeout. So we need to abort it in time. There are two optimizations. 1. Check BE's lastStartTime when get it heart beat. If current lastStartTime is large than lastStartTime in fe memory. Abort all the hang txns belong this BE. 2. Check conflict txns when do schema change. If txns is failed (Maybe coordinator be/fe restart), abort it directly.

…on case (#40765) ## Proposed changes PR #37669 introduced a new config `enable_abort_txn_by_checking_conflict_txn=true`. This config will abort schema while loading txn and restart coordinator, which causes regression test failed. We set it to false to fix this test.

github-actions bot added the meta-change label Jul 11, 2024

Lchangliang force-pushed the dev-0710-txn branch from ca83c30 to 166a237 Compare July 15, 2024 06:45

github-actions bot reviewed Jul 15, 2024

View reviewed changes

Lchangliang force-pushed the dev-0710-txn branch from 5db13f0 to 70332d6 Compare July 16, 2024 16:31

gavinchou requested changes Jul 18, 2024

View reviewed changes

luwei16 reviewed Jul 22, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/transaction/GlobalTransactionMgr.java Show resolved Hide resolved

Lchangliang force-pushed the dev-0710-txn branch from 70332d6 to caa19ef Compare July 22, 2024 09:26

gavinchou requested changes Jul 28, 2024

View reviewed changes

cloud/src/meta-service/meta_service_txn.cpp Show resolved Hide resolved

cloud/src/meta-service/meta_service_txn.cpp Show resolved Hide resolved

cloud/src/meta-service/meta_service_txn.cpp Outdated Show resolved Hide resolved

Lchangliang force-pushed the dev-0710-txn branch from 93ad64b to 7bff4b6 Compare July 29, 2024 06:16

Lchangliang changed the title ~~[draft](cloud-merge) Support to abort txn when coordinate be restart and do schema change~~ (cloud-merge) Support to abort txn when coordinate be restart and do schema change Jul 29, 2024

Lchangliang added 6 commits July 29, 2024 14:18

tmp

901aa8e

tmp

66bbb69

tmp

39d800b

tmp

f5eb42a

tmp

58cea6e

tmp

0be9160

Lchangliang force-pushed the dev-0710-txn branch from 7bff4b6 to 0be9160 Compare July 29, 2024 06:18

gavinchou reviewed Jul 29, 2024

View reviewed changes

tmp

127cd9e

gavinchou approved these changes Jul 30, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 30, 2024

github-actions bot added the reviewed label Jul 30, 2024

luwei16 approved these changes Jul 30, 2024

View reviewed changes

gavinchou merged commit 7c45912 into apache:master Jul 30, 2024

gavinchou added the dev/3.0.x label Jul 30, 2024

dataroaring added dev/3.0.2-merged and removed dev/3.0.x labels Aug 12, 2024

Yukang-Lian mentioned this pull request Sep 12, 2024

[Fix](regression test) Fix schema change concurrent with txn regression case #40765

Merged

gavinchou mentioned this pull request Oct 13, 2024

Release Note 3.0.2 #41558

Closed

(cloud-merge) Support to abort txn when coordinate be restart and do schema change #37669

(cloud-merge) Support to abort txn when coordinate be restart and do schema change #37669

Uh oh!

Conversation

Lchangliang commented Jul 11, 2024

Uh oh!

doris-robot commented Jul 11, 2024

Uh oh!

Lchangliang commented Jul 11, 2024

Uh oh!

Lchangliang commented Jul 15, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jul 15, 2024

Choose a reason for hiding this comment

Uh oh!

Lchangliang commented Jul 15, 2024

Uh oh!

Lchangliang commented Jul 15, 2024

Uh oh!

doris-robot commented Jul 15, 2024

Uh oh!

doris-robot commented Jul 15, 2024

Uh oh!

doris-robot commented Jul 15, 2024

Uh oh!

Lchangliang commented Jul 16, 2024

Uh oh!

doris-robot commented Jul 16, 2024

Uh oh!

doris-robot commented Jul 16, 2024

Uh oh!

doris-robot commented Jul 16, 2024

Uh oh!

Lchangliang commented Jul 17, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Lchangliang commented Jul 22, 2024

Uh oh!

Lchangliang commented Jul 22, 2024

Uh oh!

Lchangliang commented Jul 22, 2024

Uh oh!

doris-robot commented Jul 22, 2024

Uh oh!

doris-robot commented Jul 22, 2024

Uh oh!

doris-robot commented Jul 22, 2024

Uh oh!

Lchangliang commented Jul 25, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Lchangliang commented Jul 29, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Lchangliang commented Jul 29, 2024

Uh oh!

doris-robot commented Jul 29, 2024

Uh oh!

doris-robot commented Jul 29, 2024

Uh oh!

doris-robot commented Jul 29, 2024

Uh oh!

github-actions bot commented Jul 30, 2024

Uh oh!

github-actions bot commented Jul 30, 2024

Uh oh!

Reviewers

Assignees