[improve](binlog) Allow commit txn without waiting txn publish #48961

w41ter · 2025-03-12T08:14:32Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: selectdb/ccr-syncer#502

Problem Summary:

Commit and publish txn VS only commit txn:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

w41ter · 2025-03-12T08:14:46Z

run buildall

hello-stephen · 2025-03-12T08:14:52Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

dataroaring

LGTM

doris-robot · 2025-03-12T08:32:59Z

TeamCity cloud ut coverage result:
Function Coverage: 83.05% (1073/1292)
Line Coverage: 65.96% (17692/26824)
Region Coverage: 65.41% (8722/13334)
Branch Coverage: 55.34% (4708/8508)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c4ea02771dbd46d98fe8f74295c7b54f56cbca95_c4ea02771dbd46d98fe8f74295c7b54f56cbca95_cloud/report/index.html

github-actions · 2025-03-12T08:33:03Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-03-12T08:33:05Z

PR approved by anyone and no changes requested.

zhannngchen

LGTM

doris-robot · 2025-03-12T09:42:33Z

TPC-H: Total hot run time: 32611 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c4ea02771dbd46d98fe8f74295c7b54f56cbca95, data reload: false

------ Round 1 ----------------------------------
q1	17615	5203	5220	5203
q2	2050	285	161	161
q3	10425	1279	710	710
q4	10211	1016	554	554
q5	7752	2438	2348	2348
q6	192	172	140	140
q7	888	730	617	617
q8	9306	1280	1070	1070
q9	4929	4874	4660	4660
q10	6825	2341	1910	1910
q11	482	272	247	247
q12	339	352	217	217
q13	17784	3705	3085	3085
q14	230	229	216	216
q15	539	485	483	483
q16	623	633	569	569
q17	587	890	336	336
q18	6902	6606	6449	6449
q19	1562	959	554	554
q20	324	334	194	194
q21	2781	2168	1906	1906
q22	1066	1044	982	982
Total cold run time: 103412 ms
Total hot run time: 32611 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5262	5212	5175	5175
q2	240	331	227	227
q3	2201	2673	2287	2287
q4	1494	1828	1409	1409
q5	4241	4130	4188	4130
q6	210	167	122	122
q7	1926	1961	1769	1769
q8	2592	2665	2581	2581
q9	7281	7209	7182	7182
q10	3007	3197	2769	2769
q11	603	518	509	509
q12	721	810	601	601
q13	3516	3916	3301	3301
q14	292	301	277	277
q15	533	472	473	472
q16	667	670	659	659
q17	1153	1605	1340	1340
q18	7781	7555	7613	7555
q19	825	796	776	776
q20	1985	2036	1833	1833
q21	5532	5068	4884	4884
q22	1049	1033	999	999
Total cold run time: 53111 ms
Total hot run time: 50857 ms

doris-robot · 2025-03-12T09:54:09Z

TPC-DS: Total hot run time: 185754 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c4ea02771dbd46d98fe8f74295c7b54f56cbca95, data reload: false

query1	995	384	396	384
query2	6524	1965	1986	1965
query3	6787	217	214	214
query4	26593	23668	23411	23411
query5	4287	649	518	518
query6	304	200	194	194
query7	4608	507	288	288
query8	301	235	250	235
query9	8605	2597	2599	2597
query10	468	318	253	253
query11	15405	15281	14897	14897
query12	178	118	103	103
query13	1652	519	408	408
query14	9504	6338	6288	6288
query15	203	187	175	175
query16	7441	638	491	491
query17	1210	723	574	574
query18	1998	408	314	314
query19	197	199	166	166
query20	123	119	115	115
query21	211	128	109	109
query22	4349	4295	4054	4054
query23	34129	32977	32935	32935
query24	7650	2385	2348	2348
query25	528	461	383	383
query26	1241	267	153	153
query27	2579	468	333	333
query28	4348	2477	2414	2414
query29	768	564	418	418
query30	297	220	188	188
query31	932	846	790	790
query32	68	66	59	59
query33	559	348	304	304
query34	780	834	505	505
query35	784	811	765	765
query36	932	998	891	891
query37	116	101	75	75
query38	4214	4332	4153	4153
query39	1456	1402	1394	1394
query40	216	115	100	100
query41	55	53	51	51
query42	119	105	106	105
query43	517	507	468	468
query44	1265	784	770	770
query45	174	168	165	165
query46	823	1024	619	619
query47	1775	1808	1744	1744
query48	386	406	299	299
query49	830	495	425	425
query50	681	722	401	401
query51	4159	4180	4146	4146
query52	102	106	96	96
query53	232	245	188	188
query54	477	487	408	408
query55	80	81	80	80
query56	282	277	261	261
query57	1099	1165	1067	1067
query58	245	241	241	241
query59	2810	2897	2718	2718
query60	279	275	262	262
query61	124	121	116	116
query62	780	733	668	668
query63	228	186	184	184
query64	4339	1007	695	695
query65	4458	4333	4342	4333
query66	1117	408	298	298
query67	15897	15460	15329	15329
query68	8140	866	504	504
query69	505	308	267	267
query70	1244	1124	1107	1107
query71	462	289	256	256
query72	5779	3637	3757	3637
query73	785	729	356	356
query74	8935	9173	8812	8812
query75	3723	3177	2681	2681
query76	3679	1179	740	740
query77	778	347	291	291
query78	10129	9977	9343	9343
query79	2668	824	589	589
query80	613	511	456	456
query81	503	256	225	225
query82	737	131	96	96
query83	173	168	160	160
query84	244	98	71	71
query85	904	356	317	317
query86	389	298	295	295
query87	4726	4409	4483	4409
query88	3677	2221	2243	2221
query89	398	318	287	287
query90	1873	207	207	207
query91	143	139	109	109
query92	76	61	61	61
query93	1696	1081	572	572
query94	668	424	310	310
query95	354	269	259	259
query96	486	563	272	272
query97	3342	3358	3285	3285
query98	240	205	214	205
query99	1693	1374	1256	1256
Total cold run time: 276635 ms
Total hot run time: 185754 ms

doris-robot · 2025-03-12T09:59:32Z

ClickBench: Total hot run time: 30.91 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c4ea02771dbd46d98fe8f74295c7b54f56cbca95, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.61	0.10	0.11
query5	0.56	0.56	0.55
query6	1.17	0.72	0.73
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.51	0.52
query10	0.57	0.60	0.58
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.61	0.60
query14	2.82	2.69	2.81
query15	0.93	0.86	0.84
query16	0.39	0.38	0.39
query17	1.00	1.05	1.03
query18	0.20	0.20	0.19
query19	1.88	1.98	1.79
query20	0.02	0.01	0.01
query21	15.37	0.88	0.53
query22	0.76	1.14	0.70
query23	14.95	1.41	0.65
query24	6.98	2.00	0.91
query25	0.49	0.20	0.11
query26	0.57	0.16	0.13
query27	0.05	0.04	0.05
query28	9.72	0.78	0.43
query29	12.58	3.94	3.30
query30	0.25	0.09	0.06
query31	2.82	0.58	0.39
query32	3.23	0.55	0.46
query33	2.98	3.01	3.11
query34	15.80	5.13	4.51
query35	4.60	4.61	4.54
query36	0.66	0.49	0.47
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.13	0.13
query41	0.07	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.34 s
Total hot run time: 30.91 s

w41ter · 2025-03-12T13:40:30Z

run buildall

doris-robot · 2025-03-12T13:59:12Z

TeamCity cloud ut coverage result:
Function Coverage: 83.05% (1073/1292)
Line Coverage: 65.95% (17690/26824)
Region Coverage: 65.40% (8720/13334)
Branch Coverage: 55.24% (4700/8508)
Coverage Report: http://coverage.selectdb-in.cc/coverage/aa51be43dd66fe3506e748d72d3c36bb4295015f_aa51be43dd66fe3506e748d72d3c36bb4295015f_cloud/report/index.html

doris-robot · 2025-03-12T15:17:49Z

TPC-H: Total hot run time: 32515 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aa51be43dd66fe3506e748d72d3c36bb4295015f, data reload: false

------ Round 1 ----------------------------------
q1	17610	5165	5034	5034
q2	2039	291	169	169
q3	10442	1303	765	765
q4	10217	1032	530	530
q5	7615	2422	2399	2399
q6	193	162	130	130
q7	913	778	600	600
q8	9306	1333	1092	1092
q9	4947	4617	4560	4560
q10	6806	2320	1901	1901
q11	473	297	261	261
q12	351	363	220	220
q13	17757	3727	3111	3111
q14	232	224	205	205
q15	545	483	487	483
q16	633	614	579	579
q17	590	866	349	349
q18	7296	6568	6396	6396
q19	2193	971	596	596
q20	324	339	207	207
q21	2847	2190	1942	1942
q22	1079	1076	986	986
Total cold run time: 104408 ms
Total hot run time: 32515 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5264	5122	5126	5122
q2	243	333	232	232
q3	2138	2719	2310	2310
q4	1450	1847	1393	1393
q5	4253	4149	4147	4147
q6	212	166	125	125
q7	1958	1933	1751	1751
q8	2681	2654	2568	2568
q9	7298	7352	7258	7258
q10	3045	3244	2807	2807
q11	592	536	491	491
q12	699	748	568	568
q13	3420	4023	3340	3340
q14	298	307	264	264
q15	532	481	476	476
q16	639	675	672	672
q17	1153	1584	1366	1366
q18	7848	7673	7455	7455
q19	854	856	927	856
q20	1975	2037	1903	1903
q21	5610	5232	4822	4822
q22	1091	1071	1064	1064
Total cold run time: 53253 ms
Total hot run time: 50990 ms

doris-robot · 2025-03-12T15:30:03Z

TPC-DS: Total hot run time: 193283 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aa51be43dd66fe3506e748d72d3c36bb4295015f, data reload: false

query1	1414	1026	991	991
query2	6226	1884	1840	1840
query3	11147	4472	4639	4472
query4	53829	25757	23548	23548
query5	5207	580	502	502
query6	384	211	186	186
query7	5051	507	293	293
query8	343	257	252	252
query9	6408	2626	2640	2626
query10	413	315	254	254
query11	15417	15024	14883	14883
query12	166	119	104	104
query13	1145	529	394	394
query14	10593	6479	6661	6479
query15	232	192	190	190
query16	7041	712	495	495
query17	1074	730	574	574
query18	1507	401	333	333
query19	208	190	169	169
query20	136	132	123	123
query21	208	124	110	110
query22	4337	4620	4346	4346
query23	34244	33349	33408	33349
query24	5747	2478	2438	2438
query25	464	480	411	411
query26	688	284	157	157
query27	1794	491	334	334
query28	3130	2460	2488	2460
query29	595	586	472	472
query30	279	228	206	206
query31	895	889	801	801
query32	75	68	72	68
query33	540	357	310	310
query34	769	846	512	512
query35	811	836	766	766
query36	963	1000	911	911
query37	124	102	75	75
query38	4238	4231	4188	4188
query39	1483	1450	1445	1445
query40	211	121	106	106
query41	53	53	48	48
query42	118	109	105	105
query43	506	513	465	465
query44	1287	795	806	795
query45	180	173	177	173
query46	848	1021	664	664
query47	1842	1866	1787	1787
query48	399	439	338	338
query49	717	502	433	433
query50	712	741	415	415
query51	4283	4277	4326	4277
query52	114	118	102	102
query53	242	264	192	192
query54	519	498	433	433
query55	86	82	86	82
query56	290	276	268	268
query57	1240	1200	1175	1175
query58	253	242	243	242
query59	2624	2886	2768	2768
query60	292	282	292	282
query61	160	119	120	119
query62	755	739	679	679
query63	233	188	196	188
query64	1508	1059	698	698
query65	4585	4496	4475	4475
query66	723	411	300	300
query67	15939	15311	15251	15251
query68	5423	834	503	503
query69	516	318	287	287
query70	1176	1161	1129	1129
query71	421	309	262	262
query72	5862	3959	4017	3959
query73	1156	762	360	360
query74	9184	9230	8832	8832
query75	3240	3164	2697	2697
query76	3740	1184	771	771
query77	533	360	290	290
query78	10212	10321	9286	9286
query79	2620	841	589	589
query80	721	549	452	452
query81	495	256	231	231
query82	527	137	98	98
query83	183	173	152	152
query84	286	98	77	77
query85	826	361	316	316
query86	411	294	289	289
query87	4429	4550	4586	4550
query88	3810	2390	2291	2291
query89	401	324	288	288
query90	1756	216	217	216
query91	142	139	113	113
query92	82	60	61	60
query93	2674	1057	579	579
query94	695	415	297	297
query95	358	286	266	266
query96	500	560	275	275
query97	3373	3479	3323	3323
query98	237	207	207	207
query99	1347	1421	1311	1311
Total cold run time: 297380 ms
Total hot run time: 193283 ms

doris-robot · 2025-03-12T15:35:26Z

ClickBench: Total hot run time: 31.64 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aa51be43dd66fe3506e748d72d3c36bb4295015f, data reload: false

query1	0.04	0.04	0.03
query2	0.10	0.04	0.05
query3	0.28	0.06	0.05
query4	1.61	0.07	0.08
query5	0.58	0.56	0.55
query6	1.20	0.72	0.73
query7	0.02	0.03	0.01
query8	0.05	0.05	0.05
query9	0.63	0.54	0.54
query10	0.58	0.60	0.58
query11	0.26	0.13	0.13
query12	0.26	0.14	0.13
query13	0.63	0.62	0.63
query14	2.67	2.68	2.71
query15	1.01	0.89	0.88
query16	0.39	0.39	0.38
query17	1.03	1.03	1.07
query18	0.19	0.18	0.18
query19	1.97	2.00	1.83
query20	0.01	0.02	0.01
query21	15.37	1.00	0.67
query22	0.92	1.04	0.76
query23	14.69	1.58	0.77
query24	5.32	0.60	0.30
query25	0.17	0.09	0.08
query26	0.56	0.22	0.18
query27	0.08	0.08	0.09
query28	11.01	1.27	0.58
query29	12.55	4.22	3.46
query30	0.28	0.08	0.06
query31	2.82	0.65	0.43
query32	3.24	0.61	0.51
query33	3.04	3.10	3.20
query34	16.45	5.33	4.48
query35	4.53	4.52	4.56
query36	0.64	0.52	0.49
query37	0.20	0.18	0.17
query38	0.17	0.15	0.16
query39	0.05	0.04	0.04
query40	0.19	0.16	0.16
query41	0.11	0.05	0.05
query42	0.07	0.06	0.05
query43	0.05	0.04	0.04
Total cold run time: 106.02 s
Total hot run time: 31.64 s

dataroaring

LGTM

…e#48961) Related PR: selectdb/ccr-syncer#502 Problem Summary: Commit and publish txn VS only commit txn: ![image](https://github.com/user-attachments/assets/9c87b365-3257-48e1-aa19-c69d0610e519)

…blish #48961 (#49265) cherry pick from #48961

…blish #48961 (#49266) cherry pick from #48961

…e#48961) Related PR: selectdb/ccr-syncer#502 Problem Summary: Commit and publish txn VS only commit txn: ![image](https://github.com/user-attachments/assets/9c87b365-3257-48e1-aa19-c69d0610e519) ### Release note

w41ter requested review from dataroaring and gavinchou as code owners March 12, 2025 08:14

w41ter added dev/2.1.x dev/3.0.x labels Mar 12, 2025

dataroaring approved these changes Mar 12, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 12, 2025

github-actions bot added the reviewed label Mar 12, 2025

zhannngchen approved these changes Mar 12, 2025

View reviewed changes

[improve](binlog) Allow commit txn without waiting txn publish

aa51be4

w41ter force-pushed the commit_txn_without_wait branch from c4ea027 to aa51be4 Compare March 12, 2025 13:40

w41ter closed this Mar 13, 2025

w41ter reopened this Mar 13, 2025

dataroaring approved these changes Mar 13, 2025

View reviewed changes

dataroaring merged commit 0137cb8 into apache:master Mar 13, 2025
35 of 37 checks passed

github-actions bot added dev/3.0.x-conflict dev/2.1.x-conflict labels Mar 13, 2025

w41ter deleted the commit_txn_without_wait branch March 13, 2025 08:29

w41ter mentioned this pull request Mar 19, 2025

branch-3.0: [improve](binlog) Allow commit txn without waiting txn publish #48961 #49265

Merged

w41ter mentioned this pull request Mar 19, 2025

branch-2.1: [improve](binlog) Allow commit txn without waiting txn publish #48961 #49266

Merged

w41ter removed dev/3.0.x-conflict dev/2.1.x-conflict labels Mar 19, 2025

dataroaring pushed a commit that referenced this pull request Mar 20, 2025

branch-3.0: [improve](binlog) Allow commit txn without waiting txn pu…

555d782

…blish #48961 (#49265) cherry pick from #48961

dataroaring added dev/3.0.5-merged and removed dev/3.0.x labels Mar 20, 2025

yiguolei pushed a commit that referenced this pull request Mar 29, 2025

branch-2.1: [improve](binlog) Allow commit txn without waiting txn pu…

f550550

…blish #48961 (#49266) cherry pick from #48961

w41ter added dev/2.1.9-merged and removed dev/2.1.x labels Mar 31, 2025

yiguolei added dev/2.1.10-merged and removed dev/2.1.9-merged labels Apr 2, 2025

[improve](binlog) Allow commit txn without waiting txn publish #48961

[improve](binlog) Allow commit txn without waiting txn publish #48961

Uh oh!

Conversation

w41ter commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

w41ter commented Mar 12, 2025

Uh oh!

hello-stephen commented Mar 12, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Mar 12, 2025

Uh oh!

github-actions bot commented Mar 12, 2025

Uh oh!

github-actions bot commented Mar 12, 2025

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Mar 12, 2025

Uh oh!

doris-robot commented Mar 12, 2025

Uh oh!

doris-robot commented Mar 12, 2025

Uh oh!

w41ter commented Mar 12, 2025

Uh oh!

doris-robot commented Mar 12, 2025

Uh oh!

doris-robot commented Mar 12, 2025

Uh oh!

doris-robot commented Mar 12, 2025

Uh oh!

doris-robot commented Mar 12, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

w41ter commented Mar 12, 2025 •

edited

Loading