[Enhancement](Compaction) Make base compaction use the same tablet selection strategy as cumulative compaction #51649

Yukang-Lian · 2025-06-11T12:46:56Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Make base compaction use the same tablet selection strategy as cumulative compaction.
Modify some parameters to make them clearer

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-06-11T12:47:02Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

github-actions · 2025-06-17T09:16:12Z

PR approved by anyone and no changes requested.

Yukang-Lian · 2025-06-19T15:17:37Z

run buildall

doris-robot · 2025-06-19T15:40:53Z

TPC-H: Total hot run time: 34079 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b6b5cc45c227f9ce6c6f6e6f5e6611686ee74fd5, data reload: false

------ Round 1 ----------------------------------
q1	17601	5180	5023	5023
q2	1959	287	184	184
q3	10358	1350	725	725
q4	10300	1030	515	515
q5	9091	2372	2393	2372
q6	181	167	132	132
q7	907	743	598	598
q8	9320	1338	1140	1140
q9	6876	5097	5139	5097
q10	6893	2411	1983	1983
q11	481	304	278	278
q12	343	347	218	218
q13	17777	3646	3085	3085
q14	240	240	221	221
q15	567	478	477	477
q16	430	437	372	372
q17	613	896	360	360
q18	7618	7212	7210	7210
q19	1473	974	536	536
q20	339	339	226	226
q21	3771	2618	2364	2364
q22	1058	963	965	963
Total cold run time: 108196 ms
Total hot run time: 34079 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5148	5082	5092	5082
q2	248	324	225	225
q3	2211	2676	2308	2308
q4	1345	1783	1369	1369
q5	4269	4175	4456	4175
q6	213	171	126	126
q7	1999	1942	1767	1767
q8	2642	2545	2582	2545
q9	7175	7175	7269	7175
q10	3093	3295	2797	2797
q11	575	506	484	484
q12	668	747	615	615
q13	3574	3864	3316	3316
q14	278	305	261	261
q15	529	487	474	474
q16	424	498	441	441
q17	1178	1558	1368	1368
q18	7745	7396	7448	7396
q19	792	870	909	870
q20	1955	2049	1863	1863
q21	4952	4551	4282	4282
q22	1075	1003	969	969
Total cold run time: 52088 ms
Total hot run time: 49908 ms

doris-robot · 2025-06-19T15:52:33Z

TPC-DS: Total hot run time: 186223 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b6b5cc45c227f9ce6c6f6e6f5e6611686ee74fd5, data reload: false

query1	1008	384	411	384
query2	6547	1857	1793	1793
query3	6753	218	216	216
query4	26557	23349	23939	23349
query5	4798	630	482	482
query6	322	219	207	207
query7	4626	498	282	282
query8	275	225	217	217
query9	8639	2640	2641	2640
query10	533	336	297	297
query11	15673	15196	14868	14868
query12	170	112	107	107
query13	1655	533	432	432
query14	9585	6164	6203	6164
query15	202	205	171	171
query16	7389	643	452	452
query17	1157	725	579	579
query18	2039	409	291	291
query19	186	188	158	158
query20	120	114	115	114
query21	213	125	109	109
query22	4170	4157	3969	3969
query23	34168	33132	33203	33132
query24	8493	2363	2386	2363
query25	534	465	398	398
query26	1242	265	151	151
query27	2753	518	347	347
query28	4373	2115	2109	2109
query29	766	548	458	458
query30	292	229	194	194
query31	928	836	763	763
query32	78	65	69	65
query33	566	354	321	321
query34	782	908	540	540
query35	783	862	761	761
query36	950	1002	888	888
query37	108	100	75	75
query38	4120	4111	4134	4111
query39	1459	1446	1427	1427
query40	208	117	107	107
query41	63	61	58	58
query42	128	106	110	106
query43	505	476	478	476
query44	1323	843	825	825
query45	181	180	177	177
query46	861	1045	623	623
query47	1760	1803	1730	1730
query48	380	424	311	311
query49	750	496	401	401
query50	636	678	403	403
query51	4156	4171	4135	4135
query52	113	111	106	106
query53	223	253	182	182
query54	569	562	496	496
query55	87	89	86	86
query56	327	311	291	291
query57	1166	1197	1114	1114
query58	272	259	251	251
query59	2463	2584	2542	2542
query60	355	335	312	312
query61	123	139	151	139
query62	829	763	667	667
query63	215	188	185	185
query64	4281	1050	645	645
query65	4284	4195	4216	4195
query66	1117	410	329	329
query67	15703	15616	15426	15426
query68	9382	883	517	517
query69	470	307	268	268
query70	1181	1101	1108	1101
query71	476	327	294	294
query72	5497	4716	4717	4716
query73	715	623	354	354
query74	8908	9063	9251	9063
query75	4352	3195	2742	2742
query76	3828	1194	764	764
query77	984	376	291	291
query78	9916	10107	9231	9231
query79	4602	819	567	567
query80	685	528	528	528
query81	466	259	225	225
query82	708	132	101	101
query83	279	258	235	235
query84	298	108	96	96
query85	793	380	309	309
query86	335	303	301	301
query87	4428	4454	4298	4298
query88	2929	2293	2280	2280
query89	443	323	309	309
query90	1936	217	220	217
query91	144	218	114	114
query92	81	60	58	58
query93	2623	933	573	573
query94	669	393	307	307
query95	387	289	279	279
query96	490	570	277	277
query97	2757	2772	2634	2634
query98	234	209	209	209
query99	1406	1403	1263	1263
Total cold run time: 280492 ms
Total hot run time: 186223 ms

doris-robot · 2025-06-19T15:57:47Z

ClickBench: Total hot run time: 29.33 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b6b5cc45c227f9ce6c6f6e6f5e6611686ee74fd5, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.60	0.10	0.10
query5	0.42	0.42	0.40
query6	1.18	0.66	0.67
query7	0.03	0.02	0.01
query8	0.04	0.04	0.04
query9	0.57	0.52	0.51
query10	0.56	0.58	0.57
query11	0.16	0.10	0.11
query12	0.14	0.12	0.12
query13	0.63	0.62	0.61
query14	0.82	0.82	0.80
query15	0.91	0.87	0.87
query16	0.39	0.39	0.39
query17	1.08	1.06	1.04
query18	0.23	0.22	0.22
query19	1.99	1.90	1.89
query20	0.02	0.02	0.01
query21	15.40	0.91	0.55
query22	0.73	1.24	0.65
query23	14.93	1.37	0.60
query24	6.79	1.77	0.58
query25	0.49	0.21	0.09
query26	0.60	0.16	0.14
query27	0.06	0.06	0.04
query28	9.46	0.91	0.45
query29	12.64	4.02	3.35
query30	0.26	0.09	0.06
query31	2.83	0.62	0.39
query32	3.24	0.55	0.48
query33	3.12	3.10	3.08
query34	16.16	5.41	4.76
query35	4.82	4.83	4.85
query36	0.68	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.18	0.15	0.14
query41	0.09	0.03	0.03
query42	0.03	0.02	0.03
query43	0.04	0.02	0.03
Total cold run time: 103.84 s
Total hot run time: 29.33 s

doris-robot · 2025-06-19T17:28:22Z

BE UT Coverage Report

Increment line coverage 100.00% (12/12) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	56.36% (15053/26711)
Line Coverage	45.13% (134685/298427)
Region Coverage	44.27% (67724/152990)
Branch Coverage	38.85% (34755/89450)

hello-stephen · 2025-06-19T17:31:50Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/12) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	61.03% (16041/26285)
Line Coverage	50.51% (150649/298265)
Region Coverage	47.85% (86122/179976)
Branch Coverage	41.33% (42268/102262)

hello-stephen · 2025-06-19T17:32:05Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/12) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	61.03% (16041/26285)
Line Coverage	50.51% (150649/298265)
Region Coverage	47.85% (86122/179976)
Branch Coverage	41.33% (42268/102262)

hello-stephen · 2025-06-19T17:35:34Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/12) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	61.03% (16041/26285)
Line Coverage	50.51% (150649/298265)
Region Coverage	47.85% (86122/179976)
Branch Coverage	41.33% (42268/102262)

be/src/cloud/cloud_tablet_mgr.cpp

Yukang-Lian · 2025-06-24T09:09:38Z

run buildall

doris-robot · 2025-06-24T10:05:58Z

TPC-H: Total hot run time: 34188 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1661b1f48cc8697a88aff2c0f4e184d25efdc4cd, data reload: false

------ Round 1 ----------------------------------
q1	17580	5191	5221	5191
q2	1947	310	202	202
q3	10336	1286	729	729
q4	10228	1012	523	523
q5	7622	2466	2290	2290
q6	183	161	132	132
q7	896	746	607	607
q8	9312	1349	1189	1189
q9	6734	5094	5116	5094
q10	6892	2406	1968	1968
q11	505	287	283	283
q12	336	377	218	218
q13	17759	3700	3105	3105
q14	243	237	213	213
q15	560	504	486	486
q16	429	424	384	384
q17	600	870	387	387
q18	7574	7166	7085	7085
q19	1581	957	546	546
q20	333	334	225	225
q21	3839	3182	2366	2366
q22	1008	1012	965	965
Total cold run time: 106497 ms
Total hot run time: 34188 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5189	5120	5130	5120
q2	244	321	220	220
q3	2123	2670	2302	2302
q4	1384	1785	1360	1360
q5	4243	4390	4408	4390
q6	217	168	127	127
q7	2013	1927	1824	1824
q8	2577	2568	2495	2495
q9	7183	7108	7121	7108
q10	3047	3295	2843	2843
q11	576	524	498	498
q12	675	789	588	588
q13	3558	3934	3263	3263
q14	280	304	284	284
q15	539	481	480	480
q16	442	470	434	434
q17	1218	1559	1327	1327
q18	7354	7113	7040	7040
q19	770	798	834	798
q20	1927	1976	1816	1816
q21	4767	4397	4335	4335
q22	1054	1036	998	998
Total cold run time: 51380 ms
Total hot run time: 49650 ms

doris-robot · 2025-06-24T10:17:34Z

TPC-DS: Total hot run time: 185574 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1661b1f48cc8697a88aff2c0f4e184d25efdc4cd, data reload: false

query1	972	388	399	388
query2	6541	1801	1810	1801
query3	6735	215	220	215
query4	26406	23701	22945	22945
query5	4393	647	476	476
query6	314	213	206	206
query7	4632	511	299	299
query8	281	224	215	215
query9	8658	2609	2626	2609
query10	500	335	277	277
query11	15832	15043	15016	15016
query12	177	110	104	104
query13	1666	525	408	408
query14	9568	6242	6155	6155
query15	201	197	181	181
query16	7322	630	492	492
query17	1203	708	581	581
query18	2007	428	312	312
query19	197	206	171	171
query20	127	116	117	116
query21	216	125	108	108
query22	4142	4225	4009	4009
query23	33969	33324	32904	32904
query24	8445	2410	2427	2410
query25	554	458	390	390
query26	1231	267	151	151
query27	2742	502	344	344
query28	4303	2123	2102	2102
query29	799	546	428	428
query30	291	218	192	192
query31	951	850	771	771
query32	73	69	61	61
query33	548	376	322	322
query34	834	852	545	545
query35	792	810	716	716
query36	931	983	875	875
query37	123	100	78	78
query38	4030	4103	4131	4103
query39	1474	1436	1385	1385
query40	207	120	107	107
query41	65	60	56	56
query42	133	112	110	110
query43	497	509	477	477
query44	1383	847	817	817
query45	175	172	172	172
query46	856	1041	635	635
query47	1749	1783	1709	1709
query48	393	426	311	311
query49	751	497	375	375
query50	665	699	404	404
query51	4233	4170	4087	4087
query52	115	108	106	106
query53	231	268	186	186
query54	571	582	501	501
query55	92	87	89	87
query56	298	306	288	288
query57	1213	1193	1108	1108
query58	267	262	259	259
query59	2546	2658	2532	2532
query60	336	330	309	309
query61	127	122	136	122
query62	812	723	671	671
query63	224	189	194	189
query64	4357	1015	679	679
query65	4235	4182	4213	4182
query66	1128	415	329	329
query67	15852	15828	15313	15313
query68	8427	925	537	537
query69	463	313	280	280
query70	1196	1151	1052	1052
query71	479	330	298	298
query72	5586	4680	4737	4680
query73	715	620	358	358
query74	8923	8806	8975	8806
query75	3761	3169	2653	2653
query76	3608	1198	750	750
query77	803	425	304	304
query78	9976	10216	9516	9516
query79	2198	835	617	617
query80	628	532	442	442
query81	468	314	220	220
query82	410	128	102	102
query83	287	251	235	235
query84	292	114	86	86
query85	789	356	360	356
query86	334	322	292	292
query87	4391	4498	4369	4369
query88	2972	2294	2286	2286
query89	391	326	301	301
query90	1931	216	213	213
query91	136	142	111	111
query92	83	60	60	60
query93	1106	940	594	594
query94	678	416	316	316
query95	378	288	287	287
query96	486	575	280	280
query97	2723	2772	2687	2687
query98	235	219	206	206
query99	1479	1386	1298	1298
Total cold run time: 274023 ms
Total hot run time: 185574 ms

doris-robot · 2025-06-24T10:22:52Z

ClickBench: Total hot run time: 29.9 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1661b1f48cc8697a88aff2c0f4e184d25efdc4cd, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.04	0.03
query3	0.24	0.07	0.07
query4	1.61	0.10	0.10
query5	0.42	0.44	0.42
query6	1.17	0.66	0.66
query7	0.03	0.01	0.01
query8	0.04	0.04	0.03
query9	0.59	0.50	0.52
query10	0.57	0.58	0.57
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.63	0.61	0.61
query14	0.80	0.82	0.81
query15	0.89	0.86	0.86
query16	0.41	0.38	0.40
query17	1.04	1.04	1.07
query18	0.22	0.21	0.21
query19	1.98	1.88	1.89
query20	0.01	0.02	0.01
query21	15.37	0.91	0.53
query22	0.74	1.32	0.67
query23	14.80	1.40	0.63
query24	6.86	1.25	1.09
query25	0.48	0.32	0.08
query26	0.58	0.16	0.16
query27	0.06	0.05	0.05
query28	10.12	0.92	0.45
query29	12.57	4.00	3.32
query30	0.25	0.09	0.07
query31	2.82	0.58	0.39
query32	3.24	0.54	0.47
query33	3.13	3.10	3.08
query34	16.02	5.48	4.80
query35	4.84	4.86	4.88
query36	0.67	0.49	0.49
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.15	0.13
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 104.09 s
Total hot run time: 29.9 s

doris-robot · 2025-06-24T11:55:12Z

BE UT Coverage Report

Increment line coverage 100.00% (12/12) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	56.93% (15338/26941)
Line Coverage	46.01% (139028/302186)
Region Coverage	45.35% (70416/155264)
Branch Coverage	40.14% (37207/92686)

hello-stephen · 2025-06-24T12:55:55Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/12) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	62.21% (16488/26503)
Line Coverage	51.99% (157023/301998)
Region Coverage	49.50% (90212/182230)
Branch Coverage	43.13% (45482/105458)

hello-stephen · 2025-06-25T02:18:25Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/12) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	62.21% (16488/26503)
Line Coverage	51.99% (157023/301998)
Region Coverage	49.50% (90212/182230)
Branch Coverage	43.13% (45482/105458)

dataroaring

LGTM

github-actions · 2025-06-26T14:37:53Z

PR approved by at least one committer and no changes requested.

…lection strategy as cumulative compaction (#51649) 1. Make base compaction use the same tablet selection strategy as cumulative compaction. 2. Modify some parameters to make them clearer

…me tablet selection strategy as cumulative compaction #51649 (#52389) Cherry-picked from #51649 Co-authored-by: abmdocrt <lianyukang@selectdb.com>

…me tablet selection strategy as cumulative compaction #51649 (#52390) Cherry-picked from #51649 Co-authored-by: abmdocrt <lianyukang@selectdb.com>

…me tablet selection strategy as cumulative compaction apache#51649 (apache#52389) Cherry-picked from apache#51649 Co-authored-by: abmdocrt <lianyukang@selectdb.com>

…lection strategy as cumulative compaction apache#51649 (apache#4086) Pick doris apache#51649

…lection strategy as cumulative compaction apache#51649 (apache#4085)

…) (apache#4089) Pick from doris apache#51649

1

f2df884

Yukang-Lian requested review from dataroaring and gavinchou as code owners June 11, 2025 12:46

2

be62624

gavinchou added dev/3.0.x cloud labels Jun 12, 2025

luwei16 approved these changes Jun 17, 2025

View reviewed changes

github-actions bot added the reviewed label Jun 17, 2025

3

b6b5cc4

gavinchou reviewed Jun 24, 2025

View reviewed changes

be/src/cloud/cloud_tablet_mgr.cpp Outdated Show resolved Hide resolved

4

1661b1f

dataroaring approved these changes Jun 26, 2025

View reviewed changes

dataroaring added the dev/3.1.x label Jun 26, 2025

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 26, 2025

dataroaring merged commit bb03726 into apache:master Jun 26, 2025
24 of 26 checks passed

github-actions bot mentioned this pull request Jun 26, 2025

branch-3.0: [Enhancement](Compaction) Make base compaction use the same tablet selection strategy as cumulative compaction #51649 #52389

Merged

github-actions bot mentioned this pull request Jun 26, 2025

branch-3.1: [Enhancement](Compaction) Make base compaction use the same tablet selection strategy as cumulative compaction #51649 #52390

Merged

dataroaring added dev/3.0.7-merged and removed dev/3.0.x labels Jun 30, 2025

morrySnow added dev/3.1.0-merged and removed dev/3.1.x labels Jul 1, 2025

Hastyshell pushed a commit to Hastyshell/doris that referenced this pull request Jul 21, 2025

[Enhancement](Compaction) Make base compaction use the same tablet se…

1a906b8

…lection strategy as cumulative compaction apache#51649 (apache#4086) Pick doris apache#51649

gavinchou mentioned this pull request Aug 15, 2025

Release Note 3.0.7 #54868

Open

csun5285 pushed a commit to csun5285/doris that referenced this pull request Sep 23, 2025

[Enhancement](Compaction) Make base compaction use the same tablet se…

518f0c1

…lection strategy as cumulative compaction apache#51649 (apache#4085)

csun5285 pushed a commit to csun5285/doris that referenced this pull request Sep 23, 2025

[Enhancement](Compaction) Add some metrics for compaction (apache#51649…

8f59cd7

…) (apache#4089) Pick from doris apache#51649

[Enhancement](Compaction) Make base compaction use the same tablet selection strategy as cumulative compaction #51649

[Enhancement](Compaction) Make base compaction use the same tablet selection strategy as cumulative compaction #51649

Uh oh!

Conversation

Yukang-Lian commented Jun 11, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jun 11, 2025

Uh oh!

github-actions bot commented Jun 17, 2025

Uh oh!

Yukang-Lian commented Jun 19, 2025

Uh oh!

doris-robot commented Jun 19, 2025

Uh oh!

doris-robot commented Jun 19, 2025

Uh oh!

doris-robot commented Jun 19, 2025

Uh oh!

doris-robot commented Jun 19, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 19, 2025

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jun 19, 2025

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jun 19, 2025

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

Yukang-Lian commented Jun 24, 2025

Uh oh!

doris-robot commented Jun 24, 2025

Uh oh!

doris-robot commented Jun 24, 2025

Uh oh!

doris-robot commented Jun 24, 2025

Uh oh!

doris-robot commented Jun 24, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 24, 2025

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jun 25, 2025

BE Regression && UT Coverage Report

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 26, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants