[fix](binlog) Acquire migration lock before ingesting binlog #50663

w41ter · 2025-05-07T09:51:06Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-05-07T09:51:11Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

w41ter · 2025-05-07T09:51:39Z

run buildall

doris-robot · 2025-05-07T11:06:26Z

TPC-H: Total hot run time: 33993 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e664a8631cd54cfbf66bf0732617bbe09a5a3699, data reload: false

------ Round 1 ----------------------------------
q1	26212	4999	4952	4952
q2	2070	281	178	178
q3	10407	1227	713	713
q4	10218	1002	525	525
q5	7535	2391	2375	2375
q6	181	171	132	132
q7	923	756	612	612
q8	9329	1285	1072	1072
q9	6885	5112	5195	5112
q10	6793	2328	1883	1883
q11	501	285	275	275
q12	353	363	220	220
q13	17780	3716	3099	3099
q14	236	224	220	220
q15	531	490	490	490
q16	411	427	381	381
q17	605	866	379	379
q18	7649	7264	7134	7134
q19	1897	975	568	568
q20	339	333	225	225
q21	3954	3319	2477	2477
q22	1015	999	971	971
Total cold run time: 115824 ms
Total hot run time: 33993 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5117	5018	5048	5018
q2	238	329	227	227
q3	2166	2679	2266	2266
q4	1349	1759	1353	1353
q5	4453	4401	4403	4401
q6	223	175	130	130
q7	2008	2012	1780	1780
q8	2623	2641	2612	2612
q9	7269	7233	7032	7032
q10	2983	3205	2752	2752
q11	574	504	493	493
q12	684	774	604	604
q13	3494	3929	3208	3208
q14	298	294	301	294
q15	526	493	474	474
q16	438	471	426	426
q17	1165	1588	1396	1396
q18	7777	7654	7449	7449
q19	817	869	987	869
q20	2059	2016	1825	1825
q21	5229	4782	4757	4757
q22	1095	1088	1013	1013
Total cold run time: 52585 ms
Total hot run time: 50379 ms

doris-robot · 2025-05-07T11:18:39Z

TPC-DS: Total hot run time: 192796 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e664a8631cd54cfbf66bf0732617bbe09a5a3699, data reload: false

query1	1406	1100	1066	1066
query2	6279	1806	1811	1806
query3	11017	4553	4684	4553
query4	53484	25188	23106	23106
query5	5183	524	447	447
query6	344	207	197	197
query7	4911	500	291	291
query8	316	253	237	237
query9	5766	2580	2563	2563
query10	436	327	275	275
query11	15069	14984	14851	14851
query12	159	113	106	106
query13	1055	520	400	400
query14	10026	6455	6369	6369
query15	220	204	185	185
query16	7089	676	507	507
query17	1098	758	611	611
query18	1555	414	316	316
query19	245	185	163	163
query20	129	125	124	124
query21	202	126	111	111
query22	4526	4476	4337	4337
query23	34130	33400	33210	33210
query24	6691	2410	2413	2410
query25	449	468	414	414
query26	720	273	156	156
query27	2413	506	332	332
query28	3214	2137	2128	2128
query29	592	576	434	434
query30	272	221	204	204
query31	846	891	787	787
query32	70	61	63	61
query33	475	374	308	308
query34	779	852	546	546
query35	834	871	762	762
query36	979	1017	907	907
query37	128	102	79	79
query38	4194	4229	4249	4229
query39	1654	1409	1483	1409
query40	212	124	105	105
query41	54	52	50	50
query42	125	116	115	115
query43	489	509	488	488
query44	1316	827	825	825
query45	181	173	173	173
query46	842	1037	657	657
query47	1820	1914	1804	1804
query48	383	410	313	313
query49	693	496	434	434
query50	669	709	415	415
query51	4212	4268	4246	4246
query52	115	112	97	97
query53	223	261	184	184
query54	597	597	540	540
query55	87	89	90	89
query56	311	302	293	293
query57	1130	1241	1158	1158
query58	252	262	258	258
query59	2696	2778	2612	2612
query60	332	322	312	312
query61	132	125	124	124
query62	735	765	717	717
query63	221	185	192	185
query64	1958	1065	713	713
query65	4397	4425	4287	4287
query66	735	393	296	296
query67	15825	15845	15371	15371
query68	7270	878	508	508
query69	553	306	274	274
query70	1226	1100	1105	1100
query71	505	322	301	301
query72	5814	4828	5012	4828
query73	1414	674	345	345
query74	8926	9192	9052	9052
query75	3846	3216	2713	2713
query76	4316	1202	771	771
query77	617	373	283	283
query78	10039	10056	9327	9327
query79	2346	807	576	576
query80	640	501	527	501
query81	468	251	220	220
query82	415	123	94	94
query83	363	243	231	231
query84	290	105	87	87
query85	794	360	313	313
query86	374	305	309	305
query87	4406	4455	4281	4281
query88	3251	2205	2194	2194
query89	413	322	281	281
query90	1911	207	210	207
query91	143	143	110	110
query92	75	61	53	53
query93	1204	950	573	573
query94	682	418	307	307
query95	366	298	286	286
query96	479	566	270	270
query97	3159	3251	3092	3092
query98	232	213	202	202
query99	1414	1367	1253	1253
Total cold run time: 297648 ms
Total hot run time: 192796 ms

doris-robot · 2025-05-07T11:23:50Z

ClickBench: Total hot run time: 29.06 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e664a8631cd54cfbf66bf0732617bbe09a5a3699, data reload: false

query1	0.04	0.03	0.03
query2	0.12	0.10	0.10
query3	0.25	0.19	0.18
query4	1.59	0.19	0.19
query5	0.58	0.58	0.58
query6	1.17	0.72	0.72
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.57	0.53	0.54
query10	0.58	0.59	0.57
query11	0.16	0.11	0.11
query12	0.15	0.11	0.11
query13	0.60	0.59	0.60
query14	0.78	0.80	0.80
query15	0.87	0.84	0.86
query16	0.39	0.38	0.38
query17	1.04	1.01	1.08
query18	0.21	0.19	0.19
query19	1.95	1.84	1.81
query20	0.01	0.00	0.01
query21	15.40	0.86	0.52
query22	0.74	1.18	0.64
query23	14.95	1.38	0.67
query24	7.57	1.28	0.83
query25	0.57	0.22	0.06
query26	0.49	0.18	0.13
query27	0.06	0.05	0.04
query28	9.34	0.85	0.42
query29	12.57	3.89	3.23
query30	0.26	0.09	0.06
query31	2.81	0.59	0.39
query32	3.23	0.55	0.48
query33	3.05	3.05	3.07
query34	15.73	5.12	4.47
query35	4.45	4.52	4.48
query36	0.67	0.50	0.48
query37	0.08	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.02	0.03
query40	0.17	0.14	0.12
query41	0.08	0.03	0.03
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.5 s
Total hot run time: 29.06 s

hello-stephen · 2025-05-07T12:36:42Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	55.59% (14868/26747)
Line Coverage	44.62% (131183/294001)
Region Coverage	43.34% (66814/154157)
Branch Coverage	37.93% (33722/88896)

w41ter · 2025-05-08T02:39:45Z

run buildall

dataroaring

LGTM

github-actions · 2025-05-08T02:50:53Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-05-08T02:50:56Z

PR approved by anyone and no changes requested.

doris-robot · 2025-05-08T03:10:03Z

TPC-H: Total hot run time: 33788 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e391516d953770294f4bc6d18cba70eb88ab9106, data reload: false

------ Round 1 ----------------------------------
q1	26063	5039	4998	4998
q2	2065	277	198	198
q3	10399	1241	685	685
q4	10382	1008	520	520
q5	7723	2338	2292	2292
q6	177	176	141	141
q7	911	742	620	620
q8	9326	1246	1116	1116
q9	6949	5145	5028	5028
q10	6784	2273	1896	1896
q11	491	291	280	280
q12	338	345	210	210
q13	17783	3675	3085	3085
q14	232	233	207	207
q15	518	494	486	486
q16	421	442	374	374
q17	605	843	387	387
q18	7515	7150	7059	7059
q19	1212	940	566	566
q20	355	338	218	218
q21	4244	3305	2444	2444
q22	1045	985	978	978
Total cold run time: 115538 ms
Total hot run time: 33788 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5029	5088	5131	5088
q2	237	326	242	242
q3	2156	2656	2267	2267
q4	1406	1879	1344	1344
q5	4419	4432	4403	4403
q6	209	172	134	134
q7	2047	1918	1742	1742
q8	2614	2484	2517	2484
q9	7230	7226	7117	7117
q10	2967	3162	2737	2737
q11	570	502	504	502
q12	698	791	605	605
q13	3513	3918	3293	3293
q14	305	297	266	266
q15	511	485	473	473
q16	439	476	447	447
q17	1173	1556	1399	1399
q18	7881	7468	7598	7468
q19	814	826	834	826
q20	1939	2012	1812	1812
q21	5183	4762	4656	4656
q22	1091	1051	1008	1008
Total cold run time: 52431 ms
Total hot run time: 50313 ms

doris-robot · 2025-05-08T03:22:16Z

TPC-DS: Total hot run time: 193678 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e391516d953770294f4bc6d18cba70eb88ab9106, data reload: false

query1	1410	1090	1047	1047
query2	6171	1847	1855	1847
query3	11041	4537	4550	4537
query4	53317	25311	23647	23647
query5	5179	535	456	456
query6	324	199	189	189
query7	4884	507	281	281
query8	294	243	230	230
query9	5359	2577	2582	2577
query10	447	322	262	262
query11	15263	15111	14861	14861
query12	159	126	114	114
query13	1019	501	409	409
query14	10200	6358	6471	6358
query15	210	201	188	188
query16	7101	653	496	496
query17	1081	726	600	600
query18	1556	404	328	328
query19	196	193	162	162
query20	128	118	126	118
query21	210	135	108	108
query22	4305	4539	4363	4363
query23	34398	33653	33587	33587
query24	6996	2438	2445	2438
query25	467	476	404	404
query26	696	280	152	152
query27	2354	520	357	357
query28	3038	2113	2139	2113
query29	602	595	461	461
query30	276	225	192	192
query31	877	881	815	815
query32	75	70	63	63
query33	487	375	328	328
query34	769	858	541	541
query35	828	863	747	747
query36	952	993	930	930
query37	115	106	77	77
query38	4258	4374	4272	4272
query39	1485	1454	1465	1454
query40	212	127	115	115
query41	62	64	61	61
query42	129	115	119	115
query43	503	533	494	494
query44	1336	836	843	836
query45	189	184	176	176
query46	872	1083	644	644
query47	1827	1874	1767	1767
query48	370	422	310	310
query49	696	514	454	454
query50	659	694	417	417
query51	4194	4183	4208	4183
query52	115	110	100	100
query53	232	272	190	190
query54	599	577	515	515
query55	86	82	93	82
query56	309	312	282	282
query57	1142	1206	1142	1142
query58	285	266	267	266
query59	2744	2950	2779	2779
query60	337	336	313	313
query61	133	130	128	128
query62	738	755	689	689
query63	226	197	194	194
query64	1493	1060	732	732
query65	4284	4253	4260	4253
query66	735	398	305	305
query67	15913	15430	15332	15332
query68	7383	874	552	552
query69	540	288	256	256
query70	1195	1119	1040	1040
query71	494	308	282	282
query72	5936	4867	5099	4867
query73	1489	712	347	347
query74	8967	8897	8731	8731
query75	3894	3231	2714	2714
query76	4138	1195	740	740
query77	639	368	281	281
query78	9909	10264	9362	9362
query79	2045	776	567	567
query80	607	504	449	449
query81	474	262	226	226
query82	417	130	95	95
query83	378	249	234	234
query84	285	107	88	88
query85	810	355	317	317
query86	367	321	273	273
query87	4400	4487	4295	4295
query88	3350	2233	2221	2221
query89	405	316	281	281
query90	1944	212	224	212
query91	141	139	112	112
query92	76	62	57	57
query93	1118	915	588	588
query94	672	401	332	332
query95	374	296	301	296
query96	471	567	269	269
query97	3173	3231	3118	3118
query98	225	200	197	197
query99	1451	1401	1296	1296
Total cold run time: 296714 ms
Total hot run time: 193678 ms

doris-robot · 2025-05-08T03:27:27Z

ClickBench: Total hot run time: 29.09 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e391516d953770294f4bc6d18cba70eb88ab9106, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.10	0.10
query3	0.27	0.19	0.18
query4	1.59	0.19	0.18
query5	0.59	0.57	0.58
query6	1.17	0.71	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.58	0.54	0.51
query10	0.57	0.58	0.57
query11	0.15	0.11	0.10
query12	0.15	0.11	0.12
query13	0.61	0.60	0.60
query14	0.78	0.80	0.81
query15	0.87	0.86	0.85
query16	0.37	0.40	0.39
query17	1.02	1.07	1.02
query18	0.21	0.20	0.19
query19	1.93	1.83	1.83
query20	0.02	0.01	0.01
query21	15.42	0.87	0.54
query22	0.76	1.22	0.67
query23	14.90	1.39	0.62
query24	7.15	1.47	0.86
query25	0.45	0.25	0.08
query26	0.60	0.16	0.13
query27	0.06	0.05	0.05
query28	10.12	0.86	0.43
query29	12.60	3.93	3.23
query30	0.25	0.09	0.06
query31	2.83	0.60	0.38
query32	3.22	0.55	0.46
query33	2.99	3.01	3.05
query34	15.83	5.05	4.48
query35	4.50	4.50	4.52
query36	0.68	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.97 s
Total hot run time: 29.09 s

liaoxin01

LGTM

hello-stephen · 2025-05-08T04:13:55Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	55.61% (14874/26746)
Line Coverage	44.46% (131278/295247)
Region Coverage	43.36% (66104/152438)
Branch Coverage	38.05% (33767/88748)

w41ter · 2025-05-08T06:18:40Z

run cloud_p0

…50663)

…log #50663 (#50709) cherry pick from #50663

…log #50663 (#50708) Cherry-picked from #50663 Co-authored-by: walter <maochuan@selectdb.com>

…50663)

dataroaring added dev/2.1.x dev/3.0.x p0_l labels May 8, 2025

[fix](binlog) Acquire migration lock before ingesting binlog

e391516

w41ter force-pushed the take_migration_lock_before_ingesting branch from e664a86 to e391516 Compare May 8, 2025 02:34

dataroaring approved these changes May 8, 2025

View reviewed changes

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels May 8, 2025

liaoxin01 approved these changes May 8, 2025

View reviewed changes

dataroaring merged commit ba493a5 into apache:master May 8, 2025
23 of 26 checks passed

github-actions bot pushed a commit that referenced this pull request May 8, 2025

[fix](binlog) Acquire migration lock before ingesting binlog (#50663)

2e1f41c

github-actions bot mentioned this pull request May 8, 2025

branch-3.0: [fix](binlog) Acquire migration lock before ingesting binlog #50663 #50708

Merged

github-actions bot added the dev/2.1.x-conflict label May 8, 2025

w41ter added a commit to w41ter/incubator-doris that referenced this pull request May 8, 2025

[fix](binlog) Acquire migration lock before ingesting binlog (apache#…

4839645

…50663)

w41ter mentioned this pull request May 8, 2025

branch-2.1: [fix](binlog) Acquire migration lock before ingesting binlog #50663 #50709

Merged

w41ter removed the dev/2.1.x-conflict label May 8, 2025

w41ter deleted the take_migration_lock_before_ingesting branch May 8, 2025 09:52

yiguolei pushed a commit that referenced this pull request May 9, 2025

branch-2.1: [fix](binlog) Acquire migration lock before ingesting bin…

9422c97

…log #50663 (#50709) cherry pick from #50663

yiguolei added dev/2.1.10-merged and removed dev/2.1.x labels May 9, 2025

dataroaring pushed a commit that referenced this pull request May 9, 2025

branch-3.0: [fix](binlog) Acquire migration lock before ingesting bin…

778bb38

…log #50663 (#50708) Cherry-picked from #50663 Co-authored-by: walter <maochuan@selectdb.com>

dataroaring added dev/3.0.6-merged and removed dev/3.0.x labels May 9, 2025

yiguolei mentioned this pull request May 13, 2025

2.1.10 Release Note #50853

Closed

koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025

[fix](binlog) Acquire migration lock before ingesting binlog (apache#…

742bea7

…50663)

gavinchou mentioned this pull request Jun 11, 2025

Release Note 3.0.6 #51585

Closed

[fix](binlog) Acquire migration lock before ingesting binlog #50663

[fix](binlog) Acquire migration lock before ingesting binlog #50663

Uh oh!

Conversation

w41ter commented May 7, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented May 7, 2025

Uh oh!

w41ter commented May 7, 2025

Uh oh!

doris-robot commented May 7, 2025

Uh oh!

doris-robot commented May 7, 2025

Uh oh!

doris-robot commented May 7, 2025

Uh oh!

hello-stephen commented May 7, 2025

BE UT Coverage Report

Uh oh!

w41ter commented May 8, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 8, 2025

Uh oh!

github-actions bot commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented May 8, 2025

BE UT Coverage Report

Uh oh!

w41ter commented May 8, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants