Skip to content

[feat](binlog) Add lock binlog method #46887

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
w41ter merged 1 commit into from
Jan 17, 2025
Merged

[feat](binlog) Add lock binlog method #46887

w41ter merged 1 commit into from
Jan 17, 2025

Conversation

@w41ter
Copy link
Contributor

@w41ter w41ter commented Jan 13, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

This is the first PR for locking binlogs.

To reduce the cost of maintaining binlogs, an API named lockBinlog has been added. Users use this API to indicate which binlogs are not permitted for GC.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 13, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@w41ter
Copy link
Contributor Author

w41ter commented Jan 13, 2025

run buildall

@w41ter w41ter force-pushed the add_lock_binlog_method branch 2 times, most recently from e60ba9f to 85ec712 Compare January 14, 2025 03:38
@w41ter
Copy link
Contributor Author

w41ter commented Jan 14, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 14, 2025

TPC-H: Total hot run time: 33008 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 85ec712668b57d8bf43233f11a7f543533c55efe, data reload: false

------ Round 1 ----------------------------------
q1	17567	6175	6045	6045
q2	2055	319	168	168
q3	10592	1261	767	767
q4	10201	879	436	436
q5	7523	2192	1990	1990
q6	198	178	147	147
q7	897	753	599	599
q8	9245	1398	1163	1163
q9	5213	5079	4961	4961
q10	6769	2322	1884	1884
q11	485	281	254	254
q12	346	358	223	223
q13	17762	3771	3151	3151
q14	225	237	222	222
q15	583	541	526	526
q16	640	613	582	582
q17	589	858	318	318
q18	7120	6478	6488	6478
q19	2632	961	559	559
q20	299	312	189	189
q21	2897	2260	2029	2029
q22	374	341	317	317
Total cold run time: 104212 ms
Total hot run time: 33008 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6371	6223	6260	6223
q2	238	335	236	236
q3	2279	2716	2350	2350
q4	1460	1831	1378	1378
q5	4348	4800	4906	4800
q6	190	177	142	142
q7	2083	1973	1847	1847
q8	2658	2806	2728	2728
q9	7290	7280	7310	7280
q10	3063	3337	2728	2728
q11	595	514	510	510
q12	690	818	654	654
q13	3579	3886	3305	3305
q14	287	300	290	290
q15	596	521	519	519
q16	638	702	664	664
q17	1220	1738	1258	1258
q18	7858	7516	7443	7443
q19	802	1161	1086	1086
q20	2029	2055	1906	1906
q21	5625	5127	5170	5127
q22	659	598	562	562
Total cold run time: 54558 ms
Total hot run time: 53036 ms

@doris-robot
Copy link

doris-robot commented Jan 14, 2025

TPC-DS: Total hot run time: 195897 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 85ec712668b57d8bf43233f11a7f543533c55efe, data reload: false

query1	1298	951	925	925
query2	6227	2148	2023	2023
query3	11117	4601	4537	4537
query4	61966	29735	23575	23575
query5	5494	620	442	442
query6	414	199	177	177
query7	5553	527	293	293
query8	330	256	223	223
query9	8263	2716	2726	2716
query10	469	300	252	252
query11	17632	15090	15473	15090
query12	176	113	112	112
query13	1452	546	434	434
query14	11231	7539	6965	6965
query15	214	198	186	186
query16	7257	649	492	492
query17	1131	749	613	613
query18	1866	423	322	322
query19	206	194	171	171
query20	120	124	113	113
query21	221	128	112	112
query22	4495	4856	4301	4301
query23	33915	33650	33631	33631
query24	5650	2395	2251	2251
query25	481	474	423	423
query26	658	282	153	153
query27	1837	468	350	350
query28	4058	2520	2468	2468
query29	536	564	420	420
query30	213	189	164	164
query31	919	878	824	824
query32	93	61	62	61
query33	417	354	296	296
query34	748	884	512	512
query35	815	820	769	769
query36	997	1012	967	967
query37	149	107	79	79
query38	4281	4301	4277	4277
query39	1505	1434	1428	1428
query40	203	120	109	109
query41	52	51	53	51
query42	121	104	104	104
query43	512	536	506	506
query44	1406	858	850	850
query45	185	179	163	163
query46	896	1079	656	656
query47	1919	1876	1865	1865
query48	386	409	331	331
query49	736	500	406	406
query50	672	694	403	403
query51	7064	6957	7159	6957
query52	101	102	92	92
query53	239	264	189	189
query54	536	525	425	425
query55	87	83	78	78
query56	269	280	246	246
query57	1198	1233	1070	1070
query58	237	229	230	229
query59	3278	3088	2975	2975
query60	281	282	259	259
query61	115	115	113	113
query62	804	763	711	711
query63	224	192	190	190
query64	1286	1005	645	645
query65	3306	3170	3168	3168
query66	730	403	303	303
query67	15988	15712	15587	15587
query68	3999	837	544	544
query69	486	304	268	268
query70	1142	1120	1142	1120
query71	404	290	315	290
query72	6118	3868	3889	3868
query73	665	764	378	378
query74	9978	9169	9063	9063
query75	3186	3142	2810	2810
query76	3177	1183	761	761
query77	485	383	285	285
query78	10159	10006	9461	9461
query79	3547	836	603	603
query80	1601	542	460	460
query81	544	279	245	245
query82	659	159	117	117
query83	271	173	157	157
query84	285	101	75	75
query85	804	444	296	296
query86	459	309	306	306
query87	4480	4456	4373	4373
query88	4580	2229	2251	2229
query89	408	329	294	294
query90	1558	196	191	191
query91	141	144	111	111
query92	69	59	54	54
query93	2827	852	540	540
query94	870	391	301	301
query95	338	261	258	258
query96	485	612	299	299
query97	2800	2883	2760	2760
query98	242	201	197	197
query99	1428	1499	1355	1355
Total cold run time: 315430 ms
Total hot run time: 195897 ms

@doris-robot
Copy link

doris-robot commented Jan 14, 2025

ClickBench: Total hot run time: 31.64 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 85ec712668b57d8bf43233f11a7f543533c55efe, data reload: false

query1	0.04	0.03	0.02
query2	0.09	0.05	0.05
query3	0.22	0.06	0.06
query4	1.63	0.09	0.09
query5	0.40	0.39	0.44
query6	1.15	0.66	0.66
query7	0.03	0.02	0.01
query8	0.06	0.05	0.04
query9	0.55	0.49	0.50
query10	0.56	0.56	0.54
query11	0.16	0.12	0.12
query12	0.16	0.13	0.13
query13	0.61	0.60	0.59
query14	2.80	2.76	2.86
query15	0.91	0.83	0.84
query16	0.37	0.38	0.37
query17	1.07	1.10	1.06
query18	0.18	0.19	0.20
query19	1.88	1.86	2.10
query20	0.02	0.02	0.02
query21	15.35	0.95	0.66
query22	0.77	0.80	0.70
query23	14.90	1.48	0.68
query24	2.18	0.34	0.23
query25	0.14	0.09	0.08
query26	0.28	0.20	0.18
query27	0.09	0.08	0.08
query28	13.43	1.76	1.14
query29	12.65	4.14	3.40
query30	0.25	0.09	0.06
query31	2.82	0.60	0.40
query32	3.23	0.59	0.50
query33	3.00	3.04	3.05
query34	16.57	5.26	4.52
query35	4.60	4.61	4.62
query36	0.81	0.48	0.48
query37	0.20	0.17	0.16
query38	0.16	0.15	0.14
query39	0.05	0.04	0.04
query40	0.16	0.12	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.05	0.04
Total cold run time: 104.74 s
Total hot run time: 31.64 s

@w41ter w41ter force-pushed the add_lock_binlog_method branch from 85ec712 to 10686c9 Compare January 15, 2025 12:25
@w41ter
Copy link
Contributor Author

w41ter commented Jan 15, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 15, 2025

TPC-H: Total hot run time: 32524 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 10686c9589dc5da2890bd825c18a3c755276d298, data reload: false

------ Round 1 ----------------------------------
q1	17616	5781	5436	5436
q2	2051	323	179	179
q3	10413	1257	730	730
q4	10217	959	524	524
q5	7515	2449	2186	2186
q6	200	166	137	137
q7	920	751	623	623
q8	9238	1325	1173	1173
q9	5301	4880	4938	4880
q10	6843	2337	1892	1892
q11	487	269	251	251
q12	358	361	218	218
q13	17786	3728	3126	3126
q14	231	230	213	213
q15	527	478	484	478
q16	619	619	579	579
q17	577	892	330	330
q18	7206	6519	6559	6519
q19	1282	950	549	549
q20	318	337	194	194
q21	2831	2214	1980	1980
q22	373	332	327	327
Total cold run time: 102909 ms
Total hot run time: 32524 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5524	5650	5583	5583
q2	245	332	242	242
q3	2290	2640	2307	2307
q4	1354	1793	1340	1340
q5	4335	4796	4639	4639
q6	167	157	125	125
q7	2124	1989	1857	1857
q8	2681	2789	2681	2681
q9	7331	7193	7298	7193
q10	3003	3283	2745	2745
q11	574	531	485	485
q12	724	781	623	623
q13	3518	3886	3284	3284
q14	308	288	286	286
q15	530	458	455	455
q16	648	682	646	646
q17	1255	1738	1244	1244
q18	7768	7490	7290	7290
q19	821	1070	1127	1070
q20	2002	2014	1910	1910
q21	5709	5323	4927	4927
q22	591	579	557	557
Total cold run time: 53502 ms
Total hot run time: 51489 ms

@doris-robot
Copy link

doris-robot commented Jan 15, 2025

TPC-DS: Total hot run time: 187291 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 10686c9589dc5da2890bd825c18a3c755276d298, data reload: false

query1	973	389	375	375
query2	6529	2054	2017	2017
query3	6790	208	210	208
query4	33967	23145	23148	23145
query5	4378	646	475	475
query6	313	211	203	203
query7	4602	502	313	313
query8	284	233	229	229
query9	9472	2667	2676	2667
query10	475	306	247	247
query11	17887	15282	15049	15049
query12	157	105	103	103
query13	1646	503	385	385
query14	10302	7147	6535	6535
query15	219	199	190	190
query16	7293	636	436	436
query17	1578	724	541	541
query18	1326	373	293	293
query19	225	183	154	154
query20	117	113	109	109
query21	217	134	103	103
query22	4120	4290	4028	4028
query23	33889	33068	32910	32910
query24	6568	2318	2293	2293
query25	478	454	395	395
query26	788	278	154	154
query27	1986	479	339	339
query28	5371	2439	2422	2422
query29	546	553	436	436
query30	234	190	167	167
query31	1021	885	790	790
query32	100	62	57	57
query33	524	362	297	297
query34	757	865	496	496
query35	827	812	741	741
query36	995	1023	965	965
query37	128	105	78	78
query38	4114	4094	4111	4094
query39	1439	1403	1418	1403
query40	206	113	106	106
query41	55	55	51	51
query42	120	103	106	103
query43	519	530	497	497
query44	1299	804	804	804
query45	194	177	161	161
query46	855	1062	658	658
query47	1820	1826	1768	1768
query48	397	404	324	324
query49	774	519	428	428
query50	646	657	403	403
query51	6819	6836	6727	6727
query52	109	105	96	96
query53	231	262	195	195
query54	507	534	451	451
query55	84	78	84	78
query56	269	257	257	257
query57	1177	1149	1067	1067
query58	246	244	240	240
query59	2994	3110	2896	2896
query60	267	275	244	244
query61	121	115	119	115
query62	800	700	655	655
query63	225	194	190	190
query64	3464	1011	656	656
query65	3223	3183	3149	3149
query66	886	411	300	300
query67	15799	15684	15624	15624
query68	2875	862	551	551
query69	453	290	261	261
query70	1245	1143	1062	1062
query71	373	292	255	255
query72	5873	3860	3907	3860
query73	654	746	370	370
query74	9816	8893	8634	8634
query75	3198	3129	2671	2671
query76	2943	1189	805	805
query77	463	381	274	274
query78	10072	10151	9301	9301
query79	1014	885	600	600
query80	774	547	443	443
query81	507	278	240	240
query82	346	154	131	131
query83	191	183	162	162
query84	239	189	76	76
query85	746	356	303	303
query86	344	328	274	274
query87	4406	4478	4378	4378
query88	3703	2217	2182	2182
query89	401	329	297	297
query90	1666	196	194	194
query91	145	136	111	111
query92	59	59	60	59
query93	993	887	547	547
query94	673	418	306	306
query95	337	267	254	254
query96	500	618	298	298
query97	2775	2894	2775	2775
query98	221	202	197	197
query99	1326	1424	1335	1335
Total cold run time: 276072 ms
Total hot run time: 187291 ms

@doris-robot
Copy link

doris-robot commented Jan 15, 2025

ClickBench: Total hot run time: 30.61 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 10686c9589dc5da2890bd825c18a3c755276d298, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.61	0.11	0.10
query5	0.44	0.42	0.39
query6	1.16	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.59	0.51	0.50
query10	0.57	0.58	0.56
query11	0.14	0.10	0.11
query12	0.13	0.11	0.11
query13	0.61	0.60	0.60
query14	2.69	2.72	2.75
query15	0.91	0.84	0.83
query16	0.39	0.38	0.38
query17	1.08	1.06	1.03
query18	0.24	0.21	0.21
query19	1.87	1.86	1.99
query20	0.01	0.01	0.01
query21	15.36	0.91	0.57
query22	0.73	0.70	0.62
query23	15.46	1.38	0.60
query24	2.62	1.28	0.97
query25	0.14	0.12	0.06
query26	0.37	0.16	0.12
query27	0.08	0.04	0.06
query28	13.82	1.02	0.44
query29	12.62	4.01	3.32
query30	0.26	0.09	0.06
query31	2.83	0.60	0.38
query32	3.23	0.55	0.46
query33	2.95	3.04	3.08
query34	16.33	5.11	4.44
query35	4.46	4.46	4.47
query36	0.63	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.16	0.13	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 105.21 s
Total hot run time: 30.61 s

dataroaring
dataroaring approved these changes Jan 16, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Jan 16, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jan 16, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jan 16, 2025

PR approved by anyone and no changes requested.

Vallishp
Vallishp approved these changes Jan 16, 2025
View reviewed changes
Copy link
Contributor

@Vallishp Vallishp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@w41ter w41ter merged commit 2a55e39 into apache:master Jan 17, 2025
24 of 26 checks passed
@w41ter w41ter deleted the add_lock_binlog_method branch January 17, 2025 08:38
@w41ter w41ter mentioned this pull request Feb 6, 2025
Merged
16 tasks
dataroaring pushed a commit that referenced this pull request Feb 8, 2025
@w41ter
[feat](binlog) Speed binlog gc by locked binlogs (#47547)
6974c0a 
Related PR: #46887,
https://github.com/selectdb/ccr-syncer/pull/399/files
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Feb 19, 2025
@w41ter
[feat](binlog) Add lock binlog method (apache#46887)
d1531d9 
This is the first PR for locking binlogs. 

To reduce the cost of maintaining binlogs, an API named lockBinlog has
been added. Users use this API to indicate which binlogs are not
permitted for GC.
This was referenced Feb 19, 2025
Merged
Merged
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Feb 19, 2025
@w41ter
[feat](binlog) Add lock binlog method (apache#46887)
c907983 
This is the first PR for locking binlogs. 

To reduce the cost of maintaining binlogs, an API named lockBinlog has
been added. Users use this API to indicate which binlogs are not
permitted for GC.
dataroaring pushed a commit that referenced this pull request Feb 19, 2025
@w41ter
branch-3.0: [feat](binlog) Add lock binlog method #46887 (#48045)
d88d6f0 
cherry pick from #46887
dataroaring pushed a commit that referenced this pull request Feb 19, 2025
@w41ter
branch-2.1: [feat](binlog) Add lock binlog method #46887 (#48046)
ae07796 
cherry pick from #46887
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Feb 20, 2025
@w41ter
[feat](binlog) Speed binlog gc by locked binlogs (apache#47547)
bc274dc 
Related PR: apache#46887,
https://github.com/selectdb/ccr-syncer/pull/399/files
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Feb 20, 2025
@w41ter
[feat](binlog) Speed binlog gc by locked binlogs (apache#47547)
956ebc7 
Related PR: apache#46887,
https://github.com/selectdb/ccr-syncer/pull/399/files
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@w41ter @lzyy2024
[feat](binlog) Add lock binlog method (apache#46887)
b27bd57 
This is the first PR for locking binlogs. 

To reduce the cost of maintaining binlogs, an API named lockBinlog has
been added. Users use this API to indicate which binlogs are not
permitted for GC.
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@w41ter @lzyy2024
[feat](binlog) Speed binlog gc by locked binlogs (apache#47547)
80b02f5 
Related PR: apache#46887,
https://github.com/selectdb/ccr-syncer/pull/399/files
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@w41ter @koarz
[feat](binlog) Speed binlog gc by locked binlogs (apache#47547)
d5cba5e 
Related PR: apache#46887,
https://github.com/selectdb/ccr-syncer/pull/399/files
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Vallishp Vallishp Vallishp approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.9-merged dev/3.0.5-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@w41ter @hello-stephen @doris-robot @Vallishp @dataroaring