Skip to content

branch-3.0: [enhance](compaction) optimize mow base compaction parameters #52321 #52605

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jul 2, 2025
Merged

branch-3.0: [enhance](compaction) optimize mow base compaction parameters #52321 #52605

dataroaring merged 1 commit into from
Jul 2, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jul 1, 2025

Cherry-picked from #52321

@luwei16
[enhance](compaction) optimize mow base compaction parameters (#52321)
4abc9be 
## 1 Increase base compaction frequency
In high-frequency mow load scenarios, the default two-hour compaction
interval is too long, causing the compaction score to rise. Therefore,
adjust base_compaction_freeze_interval_s from two hours to 30 minutes.

## 2. Increase the limit on compaction score per run
The base_compaction_max_compaction_score parameter limits the number of
rowsets in a single compaction to control memory usage. However, in
high-frequency mow load scenarios, this can slow down compaction and
prevent it from keeping up with data growth. Thus, this parameter is
increased to 200.
@github-actions github-actions bot requested a review from dataroaring as a code owner July 1, 2025 13:03
@Thearas
Copy link
Contributor

Thearas commented Jul 1, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Jul 1, 2025
@dataroaring dataroaring reopened this Jul 1, 2025
@Thearas
Copy link
Contributor

Thearas commented Jul 1, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 1, 2025

TPC-H: Total hot run time: 39968 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4abc9befd7e9b4b0f9ce3d5401a1e68cfd1ef111, data reload: false

------ Round 1 ----------------------------------
q1	17676	7056	6764	6764
q2	2082	172	166	166
q3	10979	1094	1167	1094
q4	10437	768	725	725
q5	7751	2898	2895	2895
q6	218	142	136	136
q7	988	634	621	621
q8	9373	1952	2093	1952
q9	6658	6407	6445	6407
q10	7015	2254	2326	2254
q11	469	267	261	261
q12	398	210	213	210
q13	17775	3033	3011	3011
q14	243	207	209	207
q15	507	469	463	463
q16	485	404	382	382
q17	997	548	577	548
q18	7463	6513	6702	6513
q19	1402	1074	1119	1074
q20	487	204	199	199
q21	3976	3304	3117	3117
q22	1105	969	974	969
Total cold run time: 108484 ms
Total hot run time: 39968 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6717	6586	6550	6550
q2	335	240	241	240
q3	2930	2788	2924	2788
q4	2026	1813	1829	1813
q5	5817	5790	5734	5734
q6	215	131	132	131
q7	2294	1781	1818	1781
q8	3368	3574	3547	3547
q9	8972	8852	8986	8852
q10	3578	3527	3515	3515
q11	589	510	496	496
q12	823	591	602	591
q13	10526	3202	3260	3202
q14	300	256	266	256
q15	537	467	462	462
q16	491	439	446	439
q17	1853	1667	1615	1615
q18	8276	7771	7845	7771
q19	1691	1612	1639	1612
q20	2091	1814	1841	1814
q21	5226	4978	5034	4978
q22	1138	1073	1008	1008
Total cold run time: 69793 ms
Total hot run time: 59195 ms

@doris-robot
Copy link

doris-robot commented Jul 1, 2025

TPC-DS: Total hot run time: 198401 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4abc9befd7e9b4b0f9ce3d5401a1e68cfd1ef111, data reload: false

query1	1304	912	912	912
query2	6429	2007	1962	1962
query3	10819	4382	4304	4304
query4	61489	28780	23609	23609
query5	5212	462	471	462
query6	403	187	183	183
query7	5540	319	314	314
query8	318	226	222	222
query9	8924	2569	2553	2553
query10	470	273	270	270
query11	17821	15233	15736	15233
query12	164	105	104	104
query13	1469	466	427	427
query14	11151	7350	7388	7350
query15	205	185	183	183
query16	7187	476	505	476
query17	1150	608	612	608
query18	1846	330	326	326
query19	230	160	166	160
query20	126	110	110	110
query21	214	107	107	107
query22	4676	4437	4741	4437
query23	34541	34337	34327	34327
query24	6258	2963	3057	2963
query25	603	418	435	418
query26	746	178	176	176
query27	2381	372	359	359
query28	4021	2154	2120	2120
query29	703	453	435	435
query30	242	165	159	159
query31	1022	816	823	816
query32	73	55	59	55
query33	482	327	294	294
query34	921	544	547	544
query35	864	753	731	731
query36	1086	940	966	940
query37	109	64	75	64
query38	4116	4034	3988	3988
query39	1530	1464	1473	1464
query40	210	102	103	102
query41	50	48	49	48
query42	112	107	101	101
query43	531	484	495	484
query44	1203	842	816	816
query45	187	170	170	170
query46	1174	776	754	754
query47	1982	1930	1900	1900
query48	491	391	402	391
query49	765	401	386	386
query50	835	436	445	436
query51	7400	7249	7182	7182
query52	100	93	88	88
query53	268	195	200	195
query54	584	473	471	471
query55	77	80	76	76
query56	273	263	254	254
query57	1305	1185	1212	1185
query58	229	209	224	209
query59	3456	3279	3151	3151
query60	304	273	273	273
query61	124	128	120	120
query62	803	706	706	706
query63	229	201	199	199
query64	2307	677	655	655
query65	3315	3209	3220	3209
query66	757	291	294	291
query67	15933	15632	15622	15622
query68	4252	579	575	575
query69	444	265	275	265
query70	1186	1169	1096	1096
query71	334	274	262	262
query72	6378	4123	4054	4054
query73	756	357	365	357
query74	10363	8990	9076	8990
query75	3351	2671	2679	2671
query76	2069	1104	1070	1070
query77	495	289	267	267
query78	10789	9674	9673	9673
query79	2019	609	612	609
query80	1353	423	424	423
query81	512	224	216	216
query82	1249	92	92	92
query83	271	150	145	145
query84	282	89	82	82
query85	1005	325	293	293
query86	397	298	298	298
query87	4475	4259	4228	4228
query88	3525	2403	2373	2373
query89	423	300	300	300
query90	1993	191	187	187
query91	188	147	151	147
query92	63	56	51	51
query93	2883	556	541	541
query94	772	298	304	298
query95	362	281	260	260
query96	615	280	288	280
query97	3324	3171	3169	3169
query98	213	204	200	200
query99	1635	1311	1290	1290
Total cold run time: 319825 ms
Total hot run time: 198401 ms

@doris-robot
Copy link

doris-robot commented Jul 1, 2025

ClickBench: Total hot run time: 30.36 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4abc9befd7e9b4b0f9ce3d5401a1e68cfd1ef111, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.03
query3	0.24	0.06	0.07
query4	1.62	0.10	0.10
query5	0.54	0.50	0.51
query6	1.13	0.74	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.48	0.50
query10	0.55	0.56	0.54
query11	0.15	0.10	0.11
query12	0.14	0.10	0.11
query13	0.61	0.60	0.60
query14	0.79	0.80	0.81
query15	0.87	0.85	0.82
query16	0.38	0.40	0.41
query17	1.02	1.10	1.04
query18	0.23	0.21	0.21
query19	2.01	1.86	1.88
query20	0.02	0.01	0.01
query21	15.38	0.59	0.57
query22	2.28	2.01	1.79
query23	16.90	1.04	0.89
query24	3.28	1.56	1.12
query25	0.27	0.07	0.05
query26	0.51	0.13	0.14
query27	0.05	0.05	0.05
query28	9.73	0.54	0.46
query29	12.58	3.24	3.21
query30	0.24	0.06	0.06
query31	2.87	0.38	0.39
query32	3.24	0.48	0.46
query33	2.97	3.00	2.99
query34	17.01	4.47	4.48
query35	4.50	4.51	4.50
query36	0.64	0.48	0.51
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.04	0.02	0.02
query40	0.18	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 103.98 s
Total hot run time: 30.36 s

dataroaring
dataroaring approved these changes Jul 2, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 5551e58 into branch-3.0 Jul 2, 2025
22 of 25 checks passed
@github-actions github-actions bot deleted the auto-pick-52321-branch-3.0 branch July 2, 2025 02:33
koarz pushed a commit to koarz/doris that referenced this pull request Jul 3, 2025
@github-actions @luwei16 @koarz
branch-3.0: [enhance](compaction) optimize mow base compaction parame…
44fcf65 
…ters apache#52321 (apache#52605)

Cherry-picked from apache#52321

Co-authored-by: Luwei <luwei@selectdb.com>
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Thearas @doris-robot @dataroaring @luwei16