Skip to content

[enhancement](binlog) Support gc binlogs by history nums and size #34888

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
May 20, 2024
Merged

[enhancement](binlog) Support gc binlogs by history nums and size #34888

dataroaring merged 1 commit into from
May 20, 2024

Conversation

@w41ter
Copy link
Contributor

@w41ter w41ter commented May 15, 2024

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented May 15, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@w41ter
Copy link
Contributor Author

w41ter commented May 15, 2024

run buildall

@doris-robot
Copy link

doris-robot commented May 15, 2024

TPC-H: Total hot run time: 41582 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6de5f6d9eab2c76454c87b83f948a679a288ee3e, data reload: false

------ Round 1 ----------------------------------
q1	5684	4254	4266	4254
q2	390	187	181	181
q3	1420	1205	1169	1169
q4	1005	756	748	748
q5	2665	2620	2659	2620
q6	219	134	133	133
q7	1019	568	576	568
q8	1998	2119	2384	2119
q9	7017	6867	6740	6740
q10	4043	3990	3956	3956
q11	375	228	233	228
q12	394	213	226	213
q13	16443	3120	3153	3120
q14	266	229	219	219
q15	528	493	469	469
q16	481	397	386	386
q17	995	646	710	646
q18	8453	8274	7494	7494
q19	7561	1580	1570	1570
q20	688	326	326	326
q21	7630	4150	4212	4150
q22	350	281	273	273
Total cold run time: 69624 ms
Total hot run time: 41582 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4354	4232	4195	4195
q2	403	260	261	260
q3	3024	2748	2742	2742
q4	1895	1609	1572	1572
q5	5293	5305	5326	5305
q6	209	121	121	121
q7	2275	1898	1883	1883
q8	3221	3353	3345	3345
q9	8364	8428	8428	8428
q10	3895	3677	3708	3677
q11	587	484	496	484
q12	766	622	589	589
q13	3576	3026	2966	2966
q14	279	252	252	252
q15	522	469	477	469
q16	452	411	417	411
q17	1749	1505	1494	1494
q18	7809	7729	7349	7349
q19	5177	1556	1558	1556
q20	2003	1772	1765	1765
q21	5012	4821	4907	4821
q22	576	471	483	471
Total cold run time: 61441 ms
Total hot run time: 54155 ms

@doris-robot
Copy link

doris-robot commented May 15, 2024

TPC-DS: Total hot run time: 187002 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6de5f6d9eab2c76454c87b83f948a679a288ee3e, data reload: false

query1	911	391	376	376
query2	6574	2427	2218	2218
query3	6652	212	212	212
query4	23659	21271	21204	21204
query5	4133	404	413	404
query6	266	168	172	168
query7	4590	293	283	283
query8	237	187	179	179
query9	8457	2408	2388	2388
query10	434	250	243	243
query11	14734	14159	14166	14159
query12	141	90	89	89
query13	1664	384	373	373
query14	9186	8374	7662	7662
query15	223	176	177	176
query16	7162	264	266	264
query17	1030	564	561	561
query18	1924	287	280	280
query19	210	156	152	152
query20	93	93	85	85
query21	204	132	131	131
query22	5019	4767	4800	4767
query23	34495	33750	33515	33515
query24	5042	2972	2812	2812
query25	478	374	367	367
query26	687	159	153	153
query27	1830	316	332	316
query28	3786	2116	2094	2094
query29	832	603	599	599
query30	268	177	184	177
query31	917	762	741	741
query32	60	52	61	52
query33	417	250	239	239
query34	862	474	484	474
query35	764	681	666	666
query36	1072	916	883	883
query37	105	69	67	67
query38	2842	2743	2772	2743
query39	1626	1534	1552	1534
query40	196	129	122	122
query41	46	43	42	42
query42	105	96	93	93
query43	541	536	550	536
query44	1041	717	754	717
query45	258	246	259	246
query46	1075	729	720	720
query47	1937	1899	1875	1875
query48	362	303	293	293
query49	765	385	386	385
query50	770	379	391	379
query51	6796	6852	6729	6729
query52	105	85	91	85
query53	346	283	275	275
query54	509	442	418	418
query55	73	72	71	71
query56	240	217	222	217
query57	1238	1152	1163	1152
query58	210	199	199	199
query59	3361	3211	3176	3176
query60	262	259	259	259
query61	90	85	89	85
query62	569	473	470	470
query63	307	281	279	279
query64	7625	7416	7380	7380
query65	3108	3146	3118	3118
query66	735	356	343	343
query67	15475	14957	15122	14957
query68	4610	532	535	532
query69	499	303	309	303
query70	1204	1166	1161	1161
query71	428	265	272	265
query72	7972	2535	2345	2345
query73	706	341	329	329
query74	6481	6140	6072	6072
query75	3426	2679	2614	2614
query76	2378	1020	974	974
query77	658	260	259	259
query78	10558	10159	10160	10159
query79	2243	555	525	525
query80	830	424	432	424
query81	499	232	244	232
query82	693	94	94	94
query83	234	165	164	164
query84	263	87	84	84
query85	1201	262	251	251
query86	491	299	289	289
query87	3261	3139	3061	3061
query88	4092	2440	2402	2402
query89	474	388	389	388
query90	2058	192	190	190
query91	117	99	95	95
query92	58	50	48	48
query93	2320	511	493	493
query94	1202	174	180	174
query95	439	295	300	295
query96	585	280	264	264
query97	3121	2994	3044	2994
query98	232	224	208	208
query99	1231	908	887	887
Total cold run time: 266295 ms
Total hot run time: 187002 ms

@doris-robot
Copy link

doris-robot commented May 15, 2024

ClickBench: Total hot run time: 30.17 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6de5f6d9eab2c76454c87b83f948a679a288ee3e, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.08	0.09
query5	0.51	0.48	0.52
query6	1.12	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.54	0.49	0.50
query10	0.55	0.54	0.55
query11	0.16	0.12	0.11
query12	0.15	0.12	0.11
query13	0.60	0.59	0.59
query14	0.77	0.77	0.76
query15	0.83	0.80	0.82
query16	0.37	0.36	0.37
query17	1.01	0.99	1.03
query18	0.21	0.28	0.23
query19	1.86	1.74	1.69
query20	0.01	0.02	0.01
query21	15.66	0.69	0.66
query22	4.40	7.27	1.82
query23	18.36	1.46	1.21
query24	1.65	0.26	0.23
query25	0.14	0.07	0.09
query26	0.26	0.16	0.16
query27	0.08	0.07	0.08
query28	13.41	1.01	1.00
query29	13.49	3.31	3.25
query30	0.25	0.05	0.05
query31	2.86	0.38	0.38
query32	3.31	0.47	0.47
query33	2.86	2.84	2.81
query34	16.94	4.36	4.41
query35	4.50	4.45	4.50
query36	0.68	0.48	0.47
query37	0.17	0.15	0.15
query38	0.15	0.14	0.14
query39	0.05	0.03	0.03
query40	0.16	0.13	0.15
query41	0.09	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.35 s
Total hot run time: 30.17 s

@w41ter w41ter force-pushed the feature/gc_binlog_by_size_and_count branch from 6de5f6d to 7f47177 Compare May 15, 2024 06:36
@w41ter
Copy link
Contributor Author

w41ter commented May 15, 2024

run buildall

@doris-robot
Copy link

doris-robot commented May 15, 2024

TPC-H: Total hot run time: 41871 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7f471774815824514c34a7db9660e769c54f69e0, data reload: false

------ Round 1 ----------------------------------
q1	17598	4362	4241	4241
q2	2016	191	184	184
q3	10449	1235	1256	1235
q4	10213	821	812	812
q5	7484	2706	2708	2706
q6	221	129	135	129
q7	1033	604	606	604
q8	9220	2154	2077	2077
q9	9853	6724	6725	6724
q10	9235	3883	3828	3828
q11	450	248	250	248
q12	447	219	243	219
q13	17510	3138	3146	3138
q14	265	228	227	227
q15	509	477	465	465
q16	471	396	381	381
q17	983	713	724	713
q18	8406	7739	7706	7706
q19	6429	1591	1520	1520
q20	652	320	319	319
q21	5182	4113	4310	4113
q22	350	297	282	282
Total cold run time: 118976 ms
Total hot run time: 41871 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4678	4470	4454	4454
q2	374	264	262	262
q3	3182	2942	2903	2903
q4	1838	1612	1621	1612
q5	5476	5473	5467	5467
q6	220	127	133	127
q7	2317	1980	1984	1980
q8	3255	3384	3376	3376
q9	8610	8702	8593	8593
q10	3895	3790	3835	3790
q11	588	497	518	497
q12	806	629	638	629
q13	16017	3160	3216	3160
q14	305	291	278	278
q15	521	472	480	472
q16	464	402	406	402
q17	1734	1477	1473	1473
q18	7685	7552	7364	7364
q19	2267	1525	1533	1525
q20	1968	1779	1760	1760
q21	5890	4826	4785	4785
q22	604	498	495	495
Total cold run time: 72694 ms
Total hot run time: 55404 ms

@doris-robot
Copy link

doris-robot commented May 15, 2024

TPC-DS: Total hot run time: 186945 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7f471774815824514c34a7db9660e769c54f69e0, data reload: false

query1	903	389	363	363
query2	6444	2360	2451	2360
query3	6654	203	207	203
query4	23070	21050	21374	21050
query5	4141	437	437	437
query6	262	188	175	175
query7	4585	299	294	294
query8	273	200	194	194
query9	8607	2426	2408	2408
query10	443	276	255	255
query11	14750	14239	14099	14099
query12	133	91	88	88
query13	1636	384	380	380
query14	9695	7690	7937	7690
query15	225	175	171	171
query16	7893	270	270	270
query17	1712	566	594	566
query18	2004	273	271	271
query19	186	150	152	150
query20	94	83	82	82
query21	201	135	128	128
query22	5045	4888	4887	4887
query23	34316	33683	33611	33611
query24	10593	2878	2866	2866
query25	562	381	364	364
query26	697	162	151	151
query27	2212	320	321	320
query28	5621	2061	2044	2044
query29	852	612	621	612
query30	274	175	179	175
query31	982	771	746	746
query32	93	50	55	50
query33	659	247	245	245
query34	889	512	499	499
query35	789	684	687	684
query36	1097	956	912	912
query37	111	70	76	70
query38	2856	2761	2714	2714
query39	1588	1550	1567	1550
query40	198	125	125	125
query41	47	45	42	42
query42	102	96	100	96
query43	569	568	545	545
query44	1057	722	736	722
query45	265	255	251	251
query46	1065	760	741	741
query47	1948	1895	1873	1873
query48	369	303	299	299
query49	881	396	401	396
query50	753	395	397	395
query51	6800	6772	6742	6742
query52	102	87	94	87
query53	351	287	297	287
query54	856	426	437	426
query55	76	72	72	72
query56	237	217	216	216
query57	1196	1129	1141	1129
query58	215	203	198	198
query59	3299	3179	3092	3092
query60	248	230	234	230
query61	92	87	114	87
query62	652	475	475	475
query63	312	286	289	286
query64	8404	7324	7337	7324
query65	3125	3062	3076	3062
query66	775	346	343	343
query67	15506	15154	14855	14855
query68	4552	529	538	529
query69	537	340	318	318
query70	1229	1073	1159	1073
query71	404	270	268	268
query72	8138	2573	2361	2361
query73	699	332	333	332
query74	6536	6139	6126	6126
query75	3410	2673	2602	2602
query76	3106	975	911	911
query77	575	258	264	258
query78	10695	10077	9969	9969
query79	1444	527	531	527
query80	823	439	433	433
query81	537	249	244	244
query82	635	98	98	98
query83	230	162	168	162
query84	248	85	84	84
query85	1416	313	262	262
query86	461	289	291	289
query87	3284	3168	3152	3152
query88	4167	2413	2408	2408
query89	461	382	382	382
query90	1928	187	198	187
query91	137	96	99	96
query92	56	48	48	48
query93	1849	516	501	501
query94	1182	180	190	180
query95	389	312	306	306
query96	585	273	275	273
query97	3164	2962	2958	2958
query98	233	224	208	208
query99	1187	902	870	870
Total cold run time: 276944 ms
Total hot run time: 186945 ms

@doris-robot
Copy link

doris-robot commented May 15, 2024

ClickBench: Total hot run time: 29.95 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7f471774815824514c34a7db9660e769c54f69e0, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.24	0.05	0.05
query4	1.67	0.07	0.08
query5	0.49	0.49	0.53
query6	1.14	0.72	0.72
query7	0.02	0.02	0.01
query8	0.04	0.04	0.05
query9	0.54	0.48	0.49
query10	0.53	0.55	0.55
query11	0.15	0.11	0.12
query12	0.14	0.12	0.12
query13	0.59	0.58	0.59
query14	0.77	0.79	0.76
query15	0.83	0.81	0.80
query16	0.36	0.37	0.37
query17	1.01	1.01	0.95
query18	0.22	0.25	0.24
query19	1.79	1.77	1.74
query20	0.01	0.01	0.01
query21	15.63	0.66	0.65
query22	3.84	8.34	1.46
query23	18.35	1.32	1.27
query24	2.06	0.20	0.21
query25	0.13	0.09	0.08
query26	0.26	0.16	0.16
query27	0.08	0.08	0.07
query28	13.27	1.00	0.98
query29	13.84	3.35	3.34
query30	0.24	0.06	0.05
query31	2.87	0.40	0.38
query32	3.28	0.46	0.46
query33	2.86	2.81	2.92
query34	17.30	4.38	4.40
query35	4.49	4.46	4.69
query36	0.66	0.46	0.46
query37	0.17	0.16	0.15
query38	0.16	0.14	0.15
query39	0.05	0.03	0.04
query40	0.16	0.14	0.15
query41	0.09	0.04	0.05
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.54 s
Total hot run time: 29.95 s

dataroaring
dataroaring approved these changes May 15, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented May 15, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 15, 2024
@github-actions
Copy link
Contributor

github-actions bot commented May 15, 2024

PR approved by anyone and no changes requested.

@dataroaring dataroaring changed the title [feature](binlog) Support gc binlogs by history nums and size [enhancement](binlog) Support gc binlogs by history nums and size May 15, 2024
@w41ter
Copy link
Contributor Author

w41ter commented May 20, 2024

run performance

dataroaring
dataroaring approved these changes May 20, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 0c9daf1 into apache:master May 20, 2024
@w41ter w41ter deleted the feature/gc_binlog_by_size_and_count branch May 20, 2024 08:26
w41ter added a commit to w41ter/incubator-doris that referenced this pull request May 23, 2024
@w41ter
[feature](binlog) Support gc binlogs by history nums and size (apache…
8c55477 
…#34888)
@w41ter w41ter mentioned this pull request May 23, 2024
Merged
dataroaring pushed a commit that referenced this pull request May 23, 2024
@w41ter
[feature](binlog) Support gc binlogs by history nums and size (#35250)
acf741f 
* [chore](binlog) Add logs about binlog gc (#34359)

* [feature](binlog) Support gc binlogs by history nums and size (#34888)
dataroaring pushed a commit that referenced this pull request May 26, 2024
@w41ter @dataroaring
[feature](binlog) Support gc binlogs by history nums and size (#34888)
8afcbbb
w41ter added a commit to w41ter/incubator-doris that referenced this pull request May 29, 2024
@w41ter
[feature](binlog) Support gc binlogs by history nums and size
cc27962 
Cherry-pick apache#35250, refs: apache#34359, apache#34888
@w41ter w41ter mentioned this pull request May 29, 2024
Merged
dataroaring pushed a commit that referenced this pull request May 29, 2024
@w41ter
[feature](binlog) Support gc binlogs by history nums and size (#35567)
3a7f68a 
Cherry-pick #35250, refs: #34359, #34888
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@w41ter
[feature](binlog) Support gc binlogs by history nums and size (apache…
4d810f0 
…#35567)

Cherry-pick apache#35250, refs: apache#34359, apache#34888
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.4-merged dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@w41ter @doris-robot @gavinchou @dataroaring @yiguolei @wm1581066