Skip to content

branch-3.0: [Fix](Compaction) Cumulative Points Not Incrementing After Delete Operation #47282 #49732

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Apr 2, 2025
Merged

branch-3.0: [Fix](Compaction) Cumulative Points Not Incrementing After Delete Operation #47282 #49732

dataroaring merged 1 commit into from
Apr 2, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Apr 1, 2025

Cherry-picked from #47282

@Yukang-Lian
[Fix](Compaction) Cumulative Points Not Incrementing After Delete Ope…
e387724 
…ration (#47282)

When a tablet's first rowset is a delete operation followed by data rowsets, the cumulative compaction process currently skips the delete rowset and only compacts the subsequent data rowsets. This leads to a failure in incrementing the cumulative point due to the condition output_rowset->start_version() == last_cumulative_point not being satisfied.

In the current behavior, the cumulative point for the delete rowset only advances after all data rowsets have completed cumulative compaction. This creates an inefficient scenario where if continuous data ingestion occurs, all subsequent data rowsets are forced to participate in cumulative compaction, resulting in significant resource waste.

This PR fixes the issue by properly handling the cumulative point advancement when encountering delete rowsets at the beginning of the version sequence.
@github-actions github-actions bot requested a review from dataroaring as a code owner April 1, 2025 03:38
@Thearas
Copy link
Contributor

Thearas commented Apr 1, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Apr 1, 2025
@dataroaring dataroaring reopened this Apr 1, 2025
@Thearas
Copy link
Contributor

Thearas commented Apr 1, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 1, 2025

TPC-H: Total hot run time: 40283 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e3877242b23739c72f01eac66293b0efbb8f0c42, data reload: false

------ Round 1 ----------------------------------
q1	17611	6796	6600	6600
q2	2052	173	178	173
q3	10683	1069	1155	1069
q4	10574	776	729	729
q5	7759	2880	2806	2806
q6	224	134	136	134
q7	972	618	595	595
q8	9353	1944	2015	1944
q9	6656	6476	6434	6434
q10	7038	2273	2313	2273
q11	469	264	258	258
q12	402	211	215	211
q13	17760	3010	2998	2998
q14	247	209	206	206
q15	514	470	480	470
q16	679	595	599	595
q17	965	668	604	604
q18	7281	6807	6744	6744
q19	1395	1101	1129	1101
q20	486	200	202	200
q21	3915	3225	3164	3164
q22	1060	975	977	975
Total cold run time: 108095 ms
Total hot run time: 40283 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6641	6615	6609	6609
q2	333	236	230	230
q3	2866	2791	2941	2791
q4	2016	1790	1831	1790
q5	5776	5753	5725	5725
q6	214	126	136	126
q7	2171	1830	1801	1801
q8	3395	3557	3526	3526
q9	8710	8847	8761	8761
q10	3583	3558	3523	3523
q11	603	482	496	482
q12	803	589	598	589
q13	8925	3209	3193	3193
q14	299	261	266	261
q15	521	472	466	466
q16	693	652	639	639
q17	1830	1643	1611	1611
q18	8217	7703	7830	7703
q19	1696	1438	1601	1438
q20	2040	1849	1871	1849
q21	5554	5456	5349	5349
q22	1096	1043	1016	1016
Total cold run time: 67982 ms
Total hot run time: 59478 ms

@doris-robot
Copy link

doris-robot commented Apr 1, 2025

TPC-DS: Total hot run time: 197238 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e3877242b23739c72f01eac66293b0efbb8f0c42, data reload: false

query1	1330	908	916	908
query2	6222	2099	2110	2099
query3	10839	4092	4387	4092
query4	61338	28930	23292	23292
query5	5192	453	438	438
query6	392	175	169	169
query7	5457	318	303	303
query8	297	218	216	216
query9	8277	2623	2614	2614
query10	467	275	254	254
query11	17395	15207	15578	15207
query12	162	104	102	102
query13	1402	456	444	444
query14	10323	7100	6900	6900
query15	208	180	176	176
query16	7161	416	504	416
query17	1314	600	570	570
query18	1884	324	330	324
query19	219	164	174	164
query20	118	114	110	110
query21	211	105	106	105
query22	4649	4664	4589	4589
query23	34544	34014	34226	34014
query24	6156	2986	2915	2915
query25	521	402	397	397
query26	650	167	171	167
query27	1748	347	357	347
query28	4675	2446	2434	2434
query29	693	445	440	440
query30	241	172	165	165
query31	977	826	831	826
query32	71	62	64	62
query33	420	295	303	295
query34	913	518	515	515
query35	846	751	759	751
query36	1102	964	986	964
query37	119	70	78	70
query38	4111	4002	4062	4002
query39	1513	1489	1474	1474
query40	209	99	97	97
query41	48	46	47	46
query42	114	98	99	98
query43	548	513	492	492
query44	1165	845	817	817
query45	190	177	171	171
query46	1154	738	746	738
query47	2058	1977	1924	1924
query48	483	386	414	386
query49	753	414	406	406
query50	858	434	435	434
query51	7268	7327	7317	7317
query52	106	93	95	93
query53	266	188	190	188
query54	597	471	468	468
query55	81	80	78	78
query56	287	280	257	257
query57	1290	1193	1182	1182
query58	213	201	203	201
query59	3093	3008	3003	3003
query60	282	256	255	255
query61	113	110	107	107
query62	770	677	669	669
query63	211	189	186	186
query64	1424	674	632	632
query65	3258	3210	3213	3210
query66	715	295	303	295
query67	16004	15701	15652	15652
query68	4158	581	580	580
query69	437	270	276	270
query70	1144	1071	1130	1071
query71	347	255	258	255
query72	6326	4004	4009	4004
query73	761	368	342	342
query74	10359	8901	9158	8901
query75	3363	2657	2694	2657
query76	2083	1168	1111	1111
query77	485	271	274	271
query78	10540	9659	9588	9588
query79	1385	586	600	586
query80	859	441	430	430
query81	509	242	236	236
query82	1264	95	91	91
query83	253	143	147	143
query84	286	78	78	78
query85	876	296	291	291
query86	356	299	294	294
query87	4393	4291	4325	4291
query88	3520	2401	2362	2362
query89	413	286	293	286
query90	1986	186	187	186
query91	184	151	152	151
query92	72	50	48	48
query93	1625	552	544	544
query94	745	258	297	258
query95	355	254	260	254
query96	628	285	288	285
query97	3324	3184	3157	3157
query98	212	197	204	197
query99	1549	1288	1308	1288
Total cold run time: 312512 ms
Total hot run time: 197238 ms

@doris-robot
Copy link

doris-robot commented Apr 1, 2025

ClickBench: Total hot run time: 32.45 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e3877242b23739c72f01eac66293b0efbb8f0c42, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.02	0.03
query3	0.24	0.06	0.06
query4	1.61	0.11	0.10
query5	0.52	0.52	0.52
query6	1.15	0.73	0.74
query7	0.02	0.02	0.01
query8	0.03	0.05	0.04
query9	0.58	0.50	0.51
query10	0.56	0.57	0.56
query11	0.15	0.11	0.11
query12	0.14	0.12	0.11
query13	0.62	0.60	0.60
query14	2.71	2.90	2.86
query15	0.90	0.83	0.83
query16	0.38	0.40	0.38
query17	1.04	1.01	1.07
query18	0.23	0.22	0.22
query19	1.94	1.83	2.04
query20	0.01	0.01	0.01
query21	15.37	0.58	0.57
query22	2.60	2.82	2.64
query23	16.84	1.11	0.73
query24	3.18	0.41	1.96
query25	0.14	0.11	0.15
query26	0.54	0.13	0.14
query27	0.05	0.04	0.04
query28	10.03	0.55	0.46
query29	12.60	3.24	3.23
query30	0.26	0.07	0.06
query31	2.85	0.37	0.38
query32	3.26	0.45	0.46
query33	2.98	3.04	3.05
query34	17.16	4.42	4.49
query35	4.54	4.48	4.50
query36	0.68	0.47	0.49
query37	0.10	0.07	0.06
query38	0.04	0.03	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.53 s
Total hot run time: 32.45 s

@doris-robot
Copy link

doris-robot commented Apr 1, 2025

BE UT Coverage Report

Increment line coverage 75.00% (3/4) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 39.11% (10241/26188)
Line Coverage 30.44% (87294/286759)
Region Coverage 29.50% (44897/152191)
Branch Coverage 26.03% (22867/87832)

dataroaring
dataroaring approved these changes Apr 2, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor Author

github-actions bot commented Apr 2, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Apr 2, 2025
@github-actions
Copy link
Contributor Author

github-actions bot commented Apr 2, 2025

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit 2683e5b into branch-3.0 Apr 2, 2025
22 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-47282-branch-3.0 branch April 2, 2025 01:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Thearas @doris-robot @dataroaring @Yukang-Lian