Skip to content

Conversation

@hust-hhb
Copy link
Contributor

When calculating delete bitmap failed, delete bitmap lock is holded by last failed txn, fe should release lock for other txn can do commit.

Issue Number: close #xxx

Related PR: #41759

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hust-hhb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39978 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dbcf5b88fffed3ad868cb2f2e0382625d1d13cd3, data reload: false

------ Round 1 ----------------------------------
q1	17647	7522	7281	7281
q2	2057	193	176	176
q3	10628	1075	1220	1075
q4	10541	698	742	698
q5	7609	2775	2696	2696
q6	239	149	150	149
q7	966	652	605	605
q8	9243	1898	1959	1898
q9	6562	6466	6430	6430
q10	7008	2290	2359	2290
q11	484	263	258	258
q12	435	221	230	221
q13	17793	2967	2981	2967
q14	237	210	211	210
q15	580	513	516	513
q16	669	596	587	587
q17	984	598	530	530
q18	7319	6686	6721	6686
q19	1344	1080	952	952
q20	490	183	183	183
q21	4247	3306	3258	3258
q22	388	324	315	315
Total cold run time: 107470 ms
Total hot run time: 39978 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7268	7210	7207	7207
q2	328	231	232	231
q3	2975	2811	2929	2811
q4	2054	1852	1849	1849
q5	5618	5720	5678	5678
q6	226	145	147	145
q7	2265	1813	1798	1798
q8	3406	3529	3548	3529
q9	8802	8983	9010	8983
q10	3614	3566	3548	3548
q11	636	492	526	492
q12	828	626	587	587
q13	11482	3173	3072	3072
q14	324	298	272	272
q15	554	500	502	500
q16	703	651	666	651
q17	1853	1655	1638	1638
q18	8411	7743	7745	7743
q19	1729	1621	1587	1587
q20	2056	1962	1879	1879
q21	5591	5471	5422	5422
q22	636	593	583	583
Total cold run time: 71359 ms
Total hot run time: 60205 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196410 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dbcf5b88fffed3ad868cb2f2e0382625d1d13cd3, data reload: false

query1	1294	947	956	947
query2	6230	2467	2389	2389
query3	10957	4601	4724	4601
query4	33298	23304	23413	23304
query5	3771	454	447	447
query6	279	191	183	183
query7	3990	309	299	299
query8	281	242	220	220
query9	9108	2606	2601	2601
query10	453	243	237	237
query11	17893	15025	15194	15025
query12	154	105	106	105
query13	1584	440	418	418
query14	9099	7073	7151	7073
query15	262	195	192	192
query16	8231	497	503	497
query17	1617	623	606	606
query18	2166	311	342	311
query19	378	162	168	162
query20	130	114	112	112
query21	209	109	107	107
query22	4917	4590	4495	4495
query23	36178	33671	33967	33671
query24	10670	2432	2462	2432
query25	559	410	401	401
query26	849	161	159	159
query27	2658	342	333	333
query28	7979	2425	2416	2416
query29	694	453	412	412
query30	236	161	147	147
query31	1082	820	829	820
query32	95	55	56	55
query33	748	315	312	312
query34	1132	532	531	531
query35	885	757	766	757
query36	1134	962	955	955
query37	120	72	77	72
query38	4271	4455	4291	4291
query39	1495	1474	1503	1474
query40	205	104	99	99
query41	47	43	47	43
query42	119	101	104	101
query43	542	496	506	496
query44	1377	819	829	819
query45	201	162	164	162
query46	1180	735	744	735
query47	2077	1924	1926	1924
query48	408	328	318	318
query49	898	396	402	396
query50	844	411	403	403
query51	7271	7169	7166	7166
query52	105	98	101	98
query53	264	181	181	181
query54	1148	411	423	411
query55	88	79	82	79
query56	270	237	245	237
query57	1291	1148	1148	1148
query58	246	221	225	221
query59	3373	3318	3242	3242
query60	302	248	251	248
query61	110	107	106	106
query62	900	695	690	690
query63	222	194	217	194
query64	4089	687	665	665
query65	3325	3265	3248	3248
query66	718	319	312	312
query67	16452	15673	15563	15563
query68	5468	577	558	558
query69	483	245	246	245
query70	1213	1107	1136	1107
query71	457	244	266	244
query72	6927	4159	4132	4132
query73	790	355	357	355
query74	10320	8850	8919	8850
query75	3395	2636	2625	2625
query76	3783	1081	1136	1081
query77	578	268	264	264
query78	10374	9444	9383	9383
query79	1818	583	601	583
query80	733	419	517	419
query81	523	221	223	221
query82	1186	116	117	116
query83	173	146	137	137
query84	228	72	71	71
query85	1429	302	312	302
query86	445	286	295	286
query87	4463	4256	4355	4256
query88	3851	2201	2149	2149
query89	420	282	302	282
query90	1896	186	185	185
query91	138	105	103	103
query92	68	50	52	50
query93	2661	540	535	535
query94	759	283	296	283
query95	350	245	239	239
query96	625	279	272	272
query97	2866	2721	2713	2713
query98	222	191	191	191
query99	1589	1327	1299	1299
Total cold run time: 305389 ms
Total hot run time: 196410 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.36 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dbcf5b88fffed3ad868cb2f2e0382625d1d13cd3, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.07	0.07
query4	1.61	0.10	0.10
query5	0.41	0.40	0.42
query6	1.17	0.65	0.66
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.58	0.50	0.48
query10	0.55	0.58	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.61	0.59
query14	2.73	2.83	2.82
query15	0.89	0.83	0.82
query16	0.37	0.38	0.37
query17	1.07	1.06	1.05
query18	0.24	0.21	0.21
query19	1.84	1.76	1.99
query20	0.02	0.02	0.01
query21	15.36	0.60	0.58
query22	2.53	2.22	1.78
query23	17.16	0.89	0.88
query24	2.68	0.71	1.38
query25	0.19	0.08	0.12
query26	0.55	0.13	0.13
query27	0.05	0.05	0.04
query28	10.91	1.10	1.07
query29	12.62	3.31	3.27
query30	0.25	0.06	0.06
query31	2.86	0.37	0.39
query32	3.27	0.46	0.46
query33	3.16	3.16	3.06
query34	16.98	4.46	4.51
query35	4.54	4.44	4.43
query36	0.68	0.48	0.50
query37	0.10	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.03	0.02
query40	0.17	0.12	0.11
query41	0.08	0.03	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 107.06 s
Total hot run time: 32.36 s

@hust-hhb
Copy link
Contributor Author

run buildall

@hust-hhb
Copy link
Contributor Author

run buildall

@hust-hhb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39923 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 573c3f5ab2a9991287189743cc9aac1c6942086c, data reload: false

------ Round 1 ----------------------------------
q1	17588	7441	7299	7299
q2	2055	177	168	168
q3	10639	1095	1220	1095
q4	10569	722	719	719
q5	7607	2756	2675	2675
q6	241	150	152	150
q7	991	639	612	612
q8	9278	1878	1907	1878
q9	6659	6459	6476	6459
q10	7027	2321	2302	2302
q11	463	268	269	268
q12	439	224	229	224
q13	17783	2964	2887	2887
q14	243	217	220	217
q15	574	520	503	503
q16	651	581	592	581
q17	979	636	571	571
q18	7347	6661	6597	6597
q19	1341	1021	997	997
q20	449	193	188	188
q21	3979	3226	3265	3226
q22	368	312	307	307
Total cold run time: 107270 ms
Total hot run time: 39923 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7255	7232	7202	7202
q2	341	231	242	231
q3	2927	2777	2977	2777
q4	2012	1825	1787	1787
q5	5669	5639	5651	5639
q6	227	139	140	139
q7	2245	1803	1796	1796
q8	3355	3572	3509	3509
q9	8890	8976	8983	8976
q10	3617	3566	3592	3566
q11	620	512	515	512
q12	831	608	569	569
q13	12339	3158	3009	3009
q14	310	293	282	282
q15	561	497	493	493
q16	700	627	665	627
q17	1835	1620	1630	1620
q18	8211	7702	7883	7702
q19	1737	1612	1549	1549
q20	2046	1882	1922	1882
q21	5495	5429	5357	5357
q22	691	593	620	593
Total cold run time: 71914 ms
Total hot run time: 59817 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196693 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 573c3f5ab2a9991287189743cc9aac1c6942086c, data reload: false

query1	1295	943	964	943
query2	6250	2495	2365	2365
query3	10968	4776	4676	4676
query4	33535	23490	23483	23483
query5	4764	455	461	455
query6	279	175	189	175
query7	3983	309	310	309
query8	303	226	243	226
query9	9641	2775	2743	2743
query10	484	234	262	234
query11	18135	15489	15196	15196
query12	161	107	102	102
query13	1554	434	414	414
query14	10507	6843	7687	6843
query15	278	192	197	192
query16	8191	445	460	445
query17	1708	584	574	574
query18	2226	306	313	306
query19	369	157	163	157
query20	132	119	116	116
query21	208	110	114	110
query22	4850	4444	4509	4444
query23	35947	33374	33722	33374
query24	11244	2549	2466	2466
query25	638	424	422	422
query26	1349	161	154	154
query27	2935	348	342	342
query28	7741	2547	2571	2547
query29	732	420	420	420
query30	237	145	152	145
query31	1047	846	879	846
query32	89	59	56	56
query33	761	315	288	288
query34	1044	526	528	526
query35	866	796	750	750
query36	1128	959	967	959
query37	245	85	73	73
query38	4270	4201	4056	4056
query39	1549	1483	1481	1481
query40	255	106	103	103
query41	45	46	47	46
query42	119	128	103	103
query43	550	500	493	493
query44	1256	837	813	813
query45	186	171	170	170
query46	1208	714	737	714
query47	2035	1927	1914	1914
query48	428	327	332	327
query49	1053	393	398	393
query50	837	392	398	392
query51	7199	7108	7192	7108
query52	106	98	92	92
query53	259	188	184	184
query54	1168	406	418	406
query55	87	81	81	81
query56	266	254	255	254
query57	1318	1158	1170	1158
query58	248	211	217	211
query59	3449	3242	3163	3163
query60	269	263	251	251
query61	105	106	119	106
query62	928	738	747	738
query63	220	198	193	193
query64	4833	686	712	686
query65	3331	3356	3276	3276
query66	1386	304	303	303
query67	16491	15463	15508	15463
query68	6014	560	575	560
query69	477	251	250	250
query70	1194	1125	1135	1125
query71	519	254	256	254
query72	6688	4175	4164	4164
query73	817	358	368	358
query74	10157	8897	9039	8897
query75	3928	2641	2702	2641
query76	3793	1029	1099	1029
query77	648	266	272	266
query78	10229	9623	9503	9503
query79	1784	602	607	602
query80	875	428	409	409
query81	504	224	233	224
query82	413	118	124	118
query83	187	145	145	145
query84	299	72	68	68
query85	1076	299	294	294
query86	353	301	287	287
query87	4482	4437	4611	4437
query88	3817	2228	2214	2214
query89	411	292	305	292
query90	2283	184	187	184
query91	138	100	105	100
query92	64	53	50	50
query93	2607	563	560	560
query94	970	297	286	286
query95	351	245	247	245
query96	644	287	283	283
query97	2886	2672	2693	2672
query98	216	249	202	202
query99	1715	1420	1436	1420
Total cold run time: 311817 ms
Total hot run time: 196693 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.82 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 573c3f5ab2a9991287189743cc9aac1c6942086c, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.02	0.04
query3	0.24	0.07	0.07
query4	1.60	0.10	0.10
query5	0.43	0.42	0.40
query6	1.15	0.67	0.66
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.58	0.52	0.51
query10	0.55	0.58	0.55
query11	0.15	0.11	0.10
query12	0.13	0.11	0.11
query13	0.60	0.60	0.60
query14	2.85	2.73	2.72
query15	0.90	0.83	0.81
query16	0.37	0.38	0.38
query17	1.02	1.07	1.07
query18	0.24	0.22	0.21
query19	2.00	1.70	2.06
query20	0.01	0.01	0.01
query21	15.35	0.58	0.58
query22	2.61	2.16	2.08
query23	16.94	1.04	0.83
query24	3.01	0.94	0.95
query25	0.20	0.07	0.14
query26	0.50	0.14	0.14
query27	0.04	0.04	0.05
query28	10.60	1.12	1.08
query29	12.60	3.28	3.28
query30	0.25	0.07	0.06
query31	2.84	0.40	0.38
query32	3.60	0.48	0.46
query33	3.12	3.20	3.06
query34	16.93	4.47	4.48
query35	4.48	4.46	4.49
query36	0.69	0.49	0.52
query37	0.10	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.02	0.02
query40	0.18	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 107.27 s
Total hot run time: 32.82 s

@hust-hhb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39654 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 573c3f5ab2a9991287189743cc9aac1c6942086c, data reload: false

------ Round 1 ----------------------------------
q1	17599	7403	7253	7253
q2	2058	178	171	171
q3	10546	1128	1147	1128
q4	10221	747	777	747
q5	7617	2644	2677	2644
q6	244	151	143	143
q7	958	623	608	608
q8	9232	1848	1896	1848
q9	6560	6364	6419	6364
q10	6984	2401	2338	2338
q11	467	261	268	261
q12	412	221	222	221
q13	17764	2923	2921	2921
q14	248	227	212	212
q15	547	487	507	487
q16	689	591	585	585
q17	964	538	509	509
q18	7211	6617	6648	6617
q19	1349	1002	1050	1002
q20	470	186	189	186
q21	3952	3144	3095	3095
q22	381	314	318	314
Total cold run time: 106473 ms
Total hot run time: 39654 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7205	7227	7205	7205
q2	328	236	234	234
q3	2883	2761	2855	2761
q4	2134	1812	1790	1790
q5	5534	5619	5614	5614
q6	230	141	138	138
q7	2224	1813	1781	1781
q8	3278	3532	3464	3464
q9	8918	8849	8827	8827
q10	3627	3578	3545	3545
q11	598	492	491	491
q12	785	629	611	611
q13	13886	3075	3060	3060
q14	306	268	285	268
q15	551	498	542	498
q16	681	648	648	648
q17	1779	1556	1553	1553
q18	7884	7566	7580	7566
q19	1666	1618	1493	1493
q20	2036	1774	1839	1774
q21	5418	5054	5276	5054
q22	659	576	582	576
Total cold run time: 72610 ms
Total hot run time: 58951 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190121 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 573c3f5ab2a9991287189743cc9aac1c6942086c, data reload: false

query1	970	376	377	376
query2	6508	2426	2325	2325
query3	6703	212	211	211
query4	34016	23581	23381	23381
query5	4414	496	468	468
query6	280	189	192	189
query7	4626	316	318	316
query8	297	239	239	239
query9	9597	2770	2762	2762
query10	489	265	266	265
query11	18095	15183	15192	15183
query12	166	108	108	108
query13	1739	435	409	409
query14	11419	6796	7743	6796
query15	305	179	181	179
query16	8096	438	436	436
query17	1800	588	573	573
query18	2145	292	297	292
query19	363	155	151	151
query20	119	110	110	110
query21	208	128	101	101
query22	4414	4253	4082	4082
query23	34172	33319	33476	33319
query24	11412	2430	2456	2430
query25	688	394	393	393
query26	1828	153	149	149
query27	2975	324	327	324
query28	7908	2473	2468	2468
query29	1046	416	406	406
query30	298	150	146	146
query31	1028	844	786	786
query32	97	58	66	58
query33	782	349	291	291
query34	1013	521	529	521
query35	886	743	747	743
query36	1082	952	921	921
query37	286	81	77	77
query38	4196	4032	4308	4032
query39	1519	1568	1434	1434
query40	285	103	103	103
query41	46	47	47	47
query42	113	100	102	100
query43	556	509	502	502
query44	1294	810	807	807
query45	187	161	164	161
query46	1144	707	719	707
query47	1915	1835	1835	1835
query48	405	330	324	324
query49	1291	387	382	382
query50	794	383	383	383
query51	7162	7052	7052	7052
query52	108	94	93	93
query53	262	179	193	179
query54	1258	412	425	412
query55	89	79	80	79
query56	271	238	238	238
query57	1236	1110	1127	1110
query58	251	218	226	218
query59	3162	3102	3008	3008
query60	273	284	258	258
query61	109	108	112	108
query62	1031	759	776	759
query63	219	191	194	191
query64	5094	684	638	638
query65	3264	3197	3258	3197
query66	1438	308	299	299
query67	15675	15474	15479	15474
query68	6982	548	557	548
query69	451	253	255	253
query70	1225	1120	1150	1120
query71	484	250	264	250
query72	6529	4121	4026	4026
query73	800	362	375	362
query74	10514	8897	9187	8897
query75	4144	2671	2648	2648
query76	4433	1104	1031	1031
query77	765	278	359	278
query78	10197	9385	9680	9385
query79	1194	598	616	598
query80	822	449	435	435
query81	520	232	234	232
query82	617	120	119	119
query83	266	151	147	147
query84	234	75	71	71
query85	1603	313	300	300
query86	475	295	296	295
query87	4639	4500	4555	4500
query88	3651	2224	2247	2224
query89	409	308	299	299
query90	2192	192	190	190
query91	136	107	106	106
query92	71	53	56	53
query93	1707	564	564	564
query94	1074	302	285	285
query95	351	259	255	255
query96	606	293	279	279
query97	2871	2668	2637	2637
query98	212	206	196	196
query99	1690	1424	1442	1424
Total cold run time: 308781 ms
Total hot run time: 190121 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.82 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 573c3f5ab2a9991287189743cc9aac1c6942086c, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.11	0.11
query5	0.43	0.42	0.42
query6	1.14	0.67	0.65
query7	0.03	0.01	0.01
query8	0.04	0.03	0.03
query9	0.56	0.50	0.51
query10	0.56	0.57	0.57
query11	0.13	0.10	0.11
query12	0.14	0.10	0.12
query13	0.61	0.61	0.60
query14	2.87	2.83	2.73
query15	0.90	0.82	0.82
query16	0.39	0.39	0.39
query17	1.05	1.02	1.07
query18	0.24	0.21	0.21
query19	1.88	1.88	2.02
query20	0.02	0.01	0.01
query21	15.37	0.57	0.59
query22	2.78	2.58	1.86
query23	17.05	0.86	0.88
query24	3.53	1.71	1.05
query25	0.12	0.31	0.08
query26	0.55	0.14	0.14
query27	0.07	0.04	0.04
query28	9.91	1.09	1.07
query29	12.55	3.28	3.25
query30	0.25	0.07	0.06
query31	2.85	0.38	0.38
query32	3.24	0.48	0.46
query33	3.07	3.09	3.08
query34	17.02	4.41	4.51
query35	4.49	4.45	4.50
query36	0.69	0.48	0.48
query37	0.09	0.06	0.05
query38	0.05	0.03	0.03
query39	0.04	0.02	0.02
query40	0.18	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 107 s
Total hot run time: 32.82 s

@hust-hhb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32315 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 573c3f5ab2a9991287189743cc9aac1c6942086c, data reload: false

------ Round 1 ----------------------------------
q1	17593	6271	6044	6044
q2	2065	290	162	162
q3	10523	1228	735	735
q4	10238	887	439	439
q5	7884	2191	1984	1984
q6	205	179	147	147
q7	892	760	605	605
q8	9243	1329	1129	1129
q9	5188	4902	4961	4902
q10	6763	2303	1837	1837
q11	487	276	248	248
q12	352	364	213	213
q13	17762	3539	2961	2961
q14	231	233	208	208
q15	567	511	489	489
q16	621	617	580	580
q17	559	833	319	319
q18	6810	6419	6291	6291
q19	2027	960	551	551
q20	295	303	182	182
q21	2944	2126	1980	1980
q22	361	336	309	309
Total cold run time: 103610 ms
Total hot run time: 32315 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6284	6172	6170	6170
q2	233	324	232	232
q3	2214	2653	2310	2310
q4	1414	1781	1303	1303
q5	4328	4730	4766	4730
q6	195	176	144	144
q7	2045	1929	1805	1805
q8	2602	2776	2664	2664
q9	7232	7271	7371	7271
q10	3080	3329	2856	2856
q11	573	503	486	486
q12	615	730	581	581
q13	3399	3739	3131	3131
q14	286	297	261	261
q15	569	514	502	502
q16	644	693	648	648
q17	1219	1763	1253	1253
q18	7641	7546	7213	7213
q19	821	1109	1140	1109
q20	1962	2017	1915	1915
q21	5651	5240	4770	4770
q22	621	572	592	572
Total cold run time: 53628 ms
Total hot run time: 51926 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196180 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 573c3f5ab2a9991287189743cc9aac1c6942086c, data reload: false

query1	1297	985	916	916
query2	6492	2351	2266	2266
query3	10907	4610	4609	4609
query4	32948	23650	23380	23380
query5	4011	612	448	448
query6	287	201	193	193
query7	3980	503	323	323
query8	288	247	224	224
query9	9622	2737	2722	2722
query10	459	306	250	250
query11	18011	15467	15187	15187
query12	153	108	103	103
query13	1560	545	433	433
query14	10190	7269	7849	7269
query15	270	207	199	199
query16	8429	626	449	449
query17	1657	752	594	594
query18	2102	401	311	311
query19	201	179	168	168
query20	120	121	114	114
query21	212	121	108	108
query22	4550	4707	4351	4351
query23	34455	33251	34002	33251
query24	6547	2305	2368	2305
query25	513	455	405	405
query26	763	281	147	147
query27	2029	477	335	335
query28	5509	2486	2435	2435
query29	651	567	427	427
query30	226	186	147	147
query31	1010	941	823	823
query32	94	57	58	57
query33	481	350	302	302
query34	810	884	532	532
query35	818	894	778	778
query36	1039	1070	969	969
query37	126	101	81	81
query38	4209	4284	4315	4284
query39	1526	1486	1441	1441
query40	211	116	99	99
query41	44	42	45	42
query42	117	101	101	101
query43	514	526	482	482
query44	1402	829	809	809
query45	179	176	172	172
query46	897	1097	652	652
query47	2063	2036	1910	1910
query48	392	435	332	332
query49	724	477	431	431
query50	664	685	390	390
query51	7269	7118	7246	7118
query52	106	100	93	93
query53	230	257	186	186
query54	464	484	400	400
query55	83	89	79	79
query56	264	259	237	237
query57	1245	1235	1173	1173
query58	241	217	218	217
query59	3269	3225	3227	3225
query60	318	280	259	259
query61	134	149	135	135
query62	851	832	760	760
query63	231	197	191	191
query64	3369	1029	639	639
query65	3289	3274	3275	3274
query66	710	411	306	306
query67	16629	15838	15589	15589
query68	9902	784	513	513
query69	496	291	260	260
query70	1218	1153	1080	1080
query71	450	291	258	258
query72	5948	3979	3900	3900
query73	799	787	365	365
query74	10154	9314	8891	8891
query75	4582	3195	2682	2682
query76	5657	1273	789	789
query77	992	371	272	272
query78	10141	10244	9374	9374
query79	3052	930	593	593
query80	731	610	432	432
query81	496	267	233	233
query82	628	150	116	116
query83	190	163	141	141
query84	282	86	65	65
query85	764	357	303	303
query86	343	297	296	296
query87	4433	4521	4521	4521
query88	3524	2222	2217	2217
query89	432	338	309	309
query90	1811	186	186	186
query91	134	142	103	103
query92	68	53	53	53
query93	1180	901	521	521
query94	658	389	292	292
query95	327	263	245	245
query96	497	660	276	276
query97	2745	2826	2651	2651
query98	259	199	190	190
query99	1653	1558	1416	1416
Total cold run time: 299007 ms
Total hot run time: 196180 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.69 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 573c3f5ab2a9991287189743cc9aac1c6942086c, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.60	0.10	0.10
query5	0.41	0.42	0.42
query6	1.16	0.65	0.65
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.50	0.52
query10	0.54	0.56	0.56
query11	0.15	0.10	0.11
query12	0.14	0.11	0.11
query13	0.60	0.59	0.59
query14	2.80	2.88	2.76
query15	0.91	0.85	0.82
query16	0.38	0.38	0.38
query17	1.01	1.06	1.01
query18	0.24	0.21	0.21
query19	1.98	1.85	1.94
query20	0.01	0.01	0.02
query21	15.37	0.93	0.55
query22	0.77	0.93	0.77
query23	15.01	1.44	0.59
query24	2.93	1.13	2.54
query25	0.12	0.24	0.19
query26	0.25	0.15	0.14
query27	0.05	0.05	0.05
query28	14.74	1.49	1.05
query29	12.62	4.03	3.27
query30	0.25	0.10	0.07
query31	2.80	0.60	0.38
query32	3.25	0.55	0.48
query33	3.16	3.11	3.05
query34	16.82	5.11	4.50
query35	4.48	4.47	4.46
query36	0.66	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.03
query40	0.16	0.13	0.11
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.71 s
Total hot run time: 31.69 s

@hust-hhb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32397 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 821f0e1ad683369b43e8ab6c930017a2c1030bd0, data reload: false

------ Round 1 ----------------------------------
q1	17584	6138	6068	6068
q2	2051	296	168	168
q3	10435	1228	734	734
q4	10224	876	441	441
q5	7774	2165	1938	1938
q6	202	187	147	147
q7	897	766	612	612
q8	9237	1347	1131	1131
q9	5245	5072	4928	4928
q10	6760	2313	1850	1850
q11	485	276	246	246
q12	346	353	220	220
q13	17779	3591	2968	2968
q14	231	233	227	227
q15	574	507	499	499
q16	640	607	608	607
q17	571	846	324	324
q18	6944	6293	6409	6293
q19	1808	967	563	563
q20	304	317	186	186
q21	2788	2179	1942	1942
q22	360	334	305	305
Total cold run time: 103239 ms
Total hot run time: 32397 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6295	6223	6226	6223
q2	237	330	230	230
q3	2236	2688	2301	2301
q4	1392	1811	1348	1348
q5	4337	4695	4731	4695
q6	197	185	144	144
q7	2076	1873	1822	1822
q8	2677	2828	2686	2686
q9	7286	7253	7390	7253
q10	3073	3328	2844	2844
q11	586	511	489	489
q12	615	745	597	597
q13	3434	3823	3104	3104
q14	296	307	289	289
q15	578	525	513	513
q16	653	675	635	635
q17	1236	1752	1267	1267
q18	7665	7414	7332	7332
q19	816	978	1170	978
q20	1957	2044	1928	1928
q21	5761	5424	4831	4831
q22	617	594	585	585
Total cold run time: 54020 ms
Total hot run time: 52094 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196613 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 821f0e1ad683369b43e8ab6c930017a2c1030bd0, data reload: false

query1	1313	985	907	907
query2	6391	2245	2234	2234
query3	11030	4672	4584	4584
query4	33057	24232	23541	23541
query5	4736	638	467	467
query6	289	204	181	181
query7	3983	489	312	312
query8	304	248	245	245
query9	9757	2741	2744	2741
query10	505	316	255	255
query11	18107	15450	15214	15214
query12	161	108	105	105
query13	1608	542	400	400
query14	10728	7160	7777	7160
query15	250	204	184	184
query16	8404	589	467	467
query17	1516	791	586	586
query18	2107	407	304	304
query19	205	187	170	170
query20	122	115	114	114
query21	214	121	111	111
query22	4706	4697	4498	4498
query23	34611	33763	33428	33428
query24	6362	2303	2295	2295
query25	497	491	431	431
query26	816	276	161	161
query27	2089	496	328	328
query28	5493	2502	2459	2459
query29	673	589	458	458
query30	208	194	152	152
query31	979	924	851	851
query32	88	62	68	62
query33	498	416	295	295
query34	788	852	509	509
query35	829	855	742	742
query36	1005	1062	1008	1008
query37	111	101	75	75
query38	4211	4213	4183	4183
query39	1564	1477	1467	1467
query40	213	115	99	99
query41	45	42	46	42
query42	119	106	105	105
query43	515	520	495	495
query44	1385	829	804	804
query45	184	183	166	166
query46	891	1068	637	637
query47	1998	2007	1912	1912
query48	403	418	328	328
query49	720	475	392	392
query50	653	672	392	392
query51	7427	7230	7125	7125
query52	103	103	94	94
query53	228	257	190	190
query54	491	481	415	415
query55	89	78	83	78
query56	244	250	239	239
query57	1259	1259	1159	1159
query58	236	228	225	225
query59	3073	3332	3103	3103
query60	282	257	255	255
query61	123	130	113	113
query62	986	847	741	741
query63	231	189	189	189
query64	3277	1061	671	671
query65	3304	3260	3257	3257
query66	793	411	315	315
query67	16633	15906	15666	15666
query68	9888	810	546	546
query69	493	293	248	248
query70	1279	1092	1094	1092
query71	460	301	251	251
query72	5956	3828	3911	3828
query73	2042	759	357	357
query74	9830	8965	8892	8892
query75	4764	3152	2682	2682
query76	5627	1260	761	761
query77	1051	365	288	288
query78	9982	10325	9398	9398
query79	2996	911	604	604
query80	704	518	453	453
query81	476	274	224	224
query82	635	150	122	122
query83	184	163	145	145
query84	276	98	77	77
query85	793	444	296	296
query86	335	312	307	307
query87	4454	4604	4609	4604
query88	3773	2248	2216	2216
query89	425	336	297	297
query90	1836	184	193	184
query91	142	136	104	104
query92	66	54	51	51
query93	1249	904	545	545
query94	658	406	291	291
query95	345	261	250	250
query96	489	615	281	281
query97	2780	2822	2690	2690
query98	224	203	198	198
query99	1663	1543	1440	1440
Total cold run time: 302095 ms
Total hot run time: 196613 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.58 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 821f0e1ad683369b43e8ab6c930017a2c1030bd0, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.08	0.07
query4	1.62	0.10	0.11
query5	0.41	0.42	0.42
query6	1.16	0.65	0.65
query7	0.02	0.02	0.01
query8	0.03	0.04	0.03
query9	0.58	0.50	0.52
query10	0.56	0.59	0.57
query11	0.16	0.10	0.10
query12	0.14	0.10	0.10
query13	0.61	0.61	0.62
query14	2.71	2.72	2.71
query15	0.90	0.84	0.82
query16	0.39	0.37	0.38
query17	1.07	1.07	1.05
query18	0.23	0.20	0.21
query19	1.92	1.85	2.01
query20	0.01	0.02	0.01
query21	15.37	0.94	0.58
query22	0.75	0.74	0.67
query23	15.37	1.36	0.53
query24	3.13	1.23	1.87
query25	0.30	0.11	0.10
query26	0.34	0.15	0.13
query27	0.08	0.06	0.04
query28	13.87	1.49	1.04
query29	12.59	3.97	3.32
query30	0.25	0.10	0.06
query31	2.83	0.58	0.38
query32	3.23	0.56	0.47
query33	3.20	3.18	3.08
query34	16.84	5.12	4.48
query35	4.44	4.45	4.50
query36	0.66	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.09	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.61 s
Total hot run time: 31.58 s

@hust-hhb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32321 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 42b8bd2b1bafc18ae33055359e3c0a35e0ce71c9, data reload: false

------ Round 1 ----------------------------------
q1	17586	6242	6017	6017
q2	2047	305	170	170
q3	10409	1216	742	742
q4	10204	848	436	436
q5	7499	2166	1951	1951
q6	204	182	149	149
q7	904	766	606	606
q8	9228	1352	1114	1114
q9	5204	4916	4959	4916
q10	6763	2301	1876	1876
q11	478	273	247	247
q12	336	362	218	218
q13	18273	3623	2966	2966
q14	222	236	219	219
q15	563	505	492	492
q16	622	605	584	584
q17	567	836	315	315
q18	6881	6593	6340	6340
q19	1841	945	545	545
q20	304	315	183	183
q21	2871	2137	1928	1928
q22	360	338	307	307
Total cold run time: 103366 ms
Total hot run time: 32321 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6252	6223	6225	6223
q2	231	331	238	238
q3	2229	2677	2271	2271
q4	1405	1886	1331	1331
q5	4306	4742	4736	4736
q6	189	177	145	145
q7	2048	1938	1806	1806
q8	2654	2804	2678	2678
q9	7287	7209	7363	7209
q10	3079	3361	2877	2877
q11	580	511	499	499
q12	635	714	591	591
q13	3361	3774	3191	3191
q14	323	290	267	267
q15	585	493	501	493
q16	658	701	650	650
q17	1212	1770	1285	1285
q18	7762	7422	7383	7383
q19	810	1111	1100	1100
q20	2049	2053	1857	1857
q21	5774	5429	4915	4915
q22	600	630	609	609
Total cold run time: 54029 ms
Total hot run time: 52354 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197063 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 42b8bd2b1bafc18ae33055359e3c0a35e0ce71c9, data reload: false

query1	1295	1002	938	938
query2	6204	2380	2290	2290
query3	11105	4909	4901	4901
query4	33090	23712	23479	23479
query5	4700	605	466	466
query6	289	205	192	192
query7	4000	488	311	311
query8	303	244	259	244
query9	9381	2755	2763	2755
query10	488	312	251	251
query11	18054	15341	15142	15142
query12	170	111	102	102
query13	1654	561	410	410
query14	10843	8067	7356	7356
query15	240	194	186	186
query16	8068	614	460	460
query17	1502	763	587	587
query18	2022	396	321	321
query19	222	187	158	158
query20	131	117	116	116
query21	207	135	121	121
query22	4668	4642	4477	4477
query23	34682	33690	33523	33523
query24	6312	2276	2304	2276
query25	538	447	388	388
query26	793	272	151	151
query27	2130	469	330	330
query28	5531	2510	2523	2510
query29	614	556	424	424
query30	211	187	154	154
query31	1007	958	825	825
query32	73	63	62	62
query33	477	347	299	299
query34	869	894	529	529
query35	813	835	768	768
query36	1011	1078	973	973
query37	116	108	81	81
query38	4245	4262	4322	4262
query39	1529	1481	1478	1478
query40	207	118	111	111
query41	44	44	41	41
query42	123	104	106	104
query43	525	537	481	481
query44	1375	824	826	824
query45	178	177	171	171
query46	878	1067	660	660
query47	2002	2008	1978	1978
query48	377	416	322	322
query49	702	474	410	410
query50	661	677	387	387
query51	7220	7083	7170	7083
query52	106	97	94	94
query53	235	265	187	187
query54	468	495	420	420
query55	83	83	82	82
query56	253	260	238	238
query57	1259	1238	1156	1156
query58	234	239	228	228
query59	3196	3164	3120	3120
query60	269	273	264	264
query61	108	101	149	101
query62	855	822	748	748
query63	232	195	187	187
query64	3273	1020	657	657
query65	3326	3269	3260	3260
query66	701	434	308	308
query67	16496	15950	15680	15680
query68	9285	749	538	538
query69	466	306	241	241
query70	1242	1166	1093	1093
query71	423	286	260	260
query72	5902	3869	3830	3830
query73	666	744	362	362
query74	10031	9231	8893	8893
query75	4586	3192	2628	2628
query76	4732	1196	768	768
query77	885	359	268	268
query78	10157	10401	9462	9462
query79	2842	896	600	600
query80	647	524	427	427
query81	485	269	224	224
query82	629	148	129	129
query83	192	163	142	142
query84	292	98	83	83
query85	883	423	387	387
query86	350	316	309	309
query87	4428	4626	4319	4319
query88	3778	2205	2200	2200
query89	404	341	290	290
query90	1839	184	184	184
query91	147	133	103	103
query92	65	54	53	53
query93	1004	875	536	536
query94	638	404	282	282
query95	319	262	256	256
query96	483	614	283	283
query97	2752	2835	2694	2694
query98	217	200	204	200
query99	1655	1553	1421	1421
Total cold run time: 297397 ms
Total hot run time: 197063 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.79 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 42b8bd2b1bafc18ae33055359e3c0a35e0ce71c9, data reload: false

query1	0.05	0.04	0.03
query2	0.08	0.04	0.03
query3	0.24	0.06	0.07
query4	1.63	0.10	0.10
query5	0.43	0.42	0.40
query6	1.17	0.65	0.66
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.58	0.50	0.50
query10	0.55	0.58	0.55
query11	0.15	0.10	0.11
query12	0.15	0.11	0.10
query13	0.60	0.61	0.60
query14	2.70	2.76	2.77
query15	0.90	0.82	0.82
query16	0.40	0.37	0.39
query17	1.06	1.05	1.03
query18	0.22	0.21	0.21
query19	1.96	1.99	1.88
query20	0.01	0.02	0.01
query21	15.37	0.87	0.59
query22	0.75	0.71	0.70
query23	15.32	1.43	0.60
query24	3.26	1.82	1.36
query25	0.20	0.17	0.08
query26	0.25	0.15	0.15
query27	0.06	0.05	0.04
query28	14.17	1.46	1.04
query29	12.60	4.04	3.27
query30	0.25	0.09	0.06
query31	2.83	0.61	0.39
query32	3.23	0.55	0.47
query33	3.02	3.05	3.12
query34	16.70	5.14	4.47
query35	4.56	4.49	4.52
query36	0.64	0.49	0.47
query37	0.10	0.06	0.07
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.15	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 106.62 s
Total hot run time: 31.79 s

Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 31, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

commitTransaction(dbId, tableList, transactionId, tabletCommitInfos, txnCommitAttachment, false);
List<OlapTable> mowTableList = getMowTableList(tableList, tabletCommitInfos);
try {
LOG.info("try to commit transaction, transactionId: {}", transactionId);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the log ncessary?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this log record when current transaction successfully getting fe's commitLock and start do get delete bitmap lock, it's useful for checking problem

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 81e8182 into apache:master Jan 2, 2025
27 of 29 checks passed
hust-hhb added a commit to hust-hhb/doris that referenced this pull request Jan 8, 2025
zhannngchen pushed a commit that referenced this pull request Jan 10, 2025
hust-hhb added a commit to hust-hhb/doris that referenced this pull request Feb 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.4-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants