Skip to content

branch-3.1: [Opt](compaction) Prune rows with delete sign=1 in full compaction #51874 #52258

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jun 26, 2025
Merged

branch-3.1: [Opt](compaction) Prune rows with delete sign=1 in full compaction #51874 #52258

morrySnow merged 1 commit into from
Jun 26, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jun 25, 2025

Cherry-picked from #51874

@bobhan1
[Opt](compaction) Prune rows with delete sign=1 in full compaction (#…
dedccbb 
…51874)

### What problem does this PR solve?

1. Prune rows with delete sign=1 in full compaction
2. also check duplicate key in base compaction when
`enable_prune_delete_sign_when_base_compaction=false`
@github-actions github-actions bot requested a review from morrySnow as a code owner June 25, 2025 02:11
@Thearas
Copy link
Contributor

Thearas commented Jun 25, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Jun 25, 2025
@Thearas
Copy link
Contributor

Thearas commented Jun 25, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

TPC-H: Total hot run time: 39419 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dedccbb15c7bd00147d93fea4ce07034ac89ff2d, data reload: false

------ Round 1 ----------------------------------
q1	17747	6742	6557	6557
q2	2085	168	160	160
q3	10644	1084	1157	1084
q4	10226	696	708	696
q5	7741	2855	2831	2831
q6	213	135	133	133
q7	990	599	597	597
q8	9350	1936	2004	1936
q9	6667	6393	6417	6393
q10	7044	2240	2291	2240
q11	473	265	258	258
q12	400	202	205	202
q13	17783	2961	2964	2961
q14	236	205	213	205
q15	512	461	482	461
q16	476	384	376	376
q17	964	528	611	528
q18	7354	6591	6693	6591
q19	1332	899	945	899
q20	485	207	206	206
q21	4147	3146	3127	3127
q22	1111	1018	978	978
Total cold run time: 107980 ms
Total hot run time: 39419 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6562	6600	6533	6533
q2	330	234	230	230
q3	2891	2767	2754	2754
q4	2077	1767	1795	1767
q5	5723	5764	5691	5691
q6	212	126	126	126
q7	2202	1793	1803	1793
q8	3385	3561	3542	3542
q9	8918	8690	8918	8690
q10	3570	3502	3514	3502
q11	584	499	487	487
q12	809	589	603	589
q13	7679	3146	3172	3146
q14	292	256	269	256
q15	517	464	470	464
q16	486	426	435	426
q17	1847	1616	1590	1590
q18	8176	7810	7727	7727
q19	1688	1482	1538	1482
q20	2139	1799	1815	1799
q21	5118	4939	4985	4939
q22	1133	1019	1047	1019
Total cold run time: 66338 ms
Total hot run time: 58552 ms

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

TPC-DS: Total hot run time: 197081 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dedccbb15c7bd00147d93fea4ce07034ac89ff2d, data reload: false

query1	1275	920	885	885
query2	6321	1900	1850	1850
query3	10841	4278	4296	4278
query4	60927	28755	23496	23496
query5	5160	460	441	441
query6	418	179	196	179
query7	5463	325	305	305
query8	310	238	238	238
query9	8499	2606	2578	2578
query10	495	274	269	269
query11	17562	15295	15710	15295
query12	165	115	105	105
query13	1440	425	436	425
query14	10269	7175	7079	7079
query15	197	178	186	178
query16	7135	505	491	491
query17	1144	588	589	588
query18	1835	339	319	319
query19	205	159	156	156
query20	114	117	113	113
query21	218	109	102	102
query22	4768	4376	4784	4376
query23	34707	34580	34485	34485
query24	6284	2953	2949	2949
query25	553	420	419	419
query26	655	170	174	170
query27	1825	365	355	355
query28	3988	2203	2139	2139
query29	719	461	450	450
query30	252	159	157	157
query31	1014	819	787	787
query32	69	57	59	57
query33	482	308	317	308
query34	924	519	525	519
query35	816	738	752	738
query36	1065	982	986	982
query37	117	74	68	68
query38	4109	3995	3906	3906
query39	1532	1459	1467	1459
query40	194	100	94	94
query41	48	46	45	45
query42	116	104	96	96
query43	507	477	476	476
query44	1203	819	826	819
query45	183	172	168	168
query46	1184	718	762	718
query47	2072	1879	1944	1879
query48	422	339	346	339
query49	750	392	397	392
query50	862	431	437	431
query51	7563	7292	7211	7211
query52	108	95	94	94
query53	262	179	178	178
query54	579	477	469	469
query55	85	81	81	81
query56	262	245	234	234
query57	1334	1190	1214	1190
query58	228	211	239	211
query59	3154	2993	3052	2993
query60	298	270	271	270
query61	114	115	111	111
query62	825	714	676	676
query63	218	187	191	187
query64	2084	658	628	628
query65	3256	3223	3165	3165
query66	747	293	319	293
query67	15868	15558	15406	15406
query68	4185	584	572	572
query69	438	273	263	263
query70	1165	1125	1141	1125
query71	340	254	257	254
query72	6415	4033	4153	4033
query73	749	347	354	347
query74	10470	9244	9069	9069
query75	3366	2653	2653	2653
query76	1927	1058	1073	1058
query77	490	276	267	267
query78	10633	9646	9533	9533
query79	2095	602	597	597
query80	1346	429	419	419
query81	516	231	218	218
query82	1257	88	87	87
query83	270	144	150	144
query84	282	83	83	83
query85	1001	291	298	291
query86	401	304	298	298
query87	4406	4277	4243	4243
query88	3966	2394	2344	2344
query89	423	297	291	291
query90	1963	185	187	185
query91	144	110	106	106
query92	66	50	50	50
query93	2914	551	551	551
query94	803	302	307	302
query95	362	261	261	261
query96	625	283	282	282
query97	3305	3146	3134	3134
query98	254	196	196	196
query99	1606	1299	1277	1277
Total cold run time: 316581 ms
Total hot run time: 197081 ms

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

ClickBench: Total hot run time: 30.38 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dedccbb15c7bd00147d93fea4ce07034ac89ff2d, data reload: false

query1	0.03	0.02	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.62	0.10	0.10
query5	0.50	0.52	0.52
query6	1.13	0.76	0.73
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.55	0.50	0.50
query10	0.56	0.56	0.55
query11	0.14	0.10	0.10
query12	0.14	0.12	0.11
query13	0.61	0.59	0.59
query14	0.78	0.77	0.81
query15	0.84	0.82	0.82
query16	0.38	0.38	0.36
query17	1.01	1.05	0.99
query18	0.24	0.23	0.23
query19	1.94	1.89	1.90
query20	0.02	0.01	0.00
query21	15.38	0.60	0.59
query22	2.27	1.92	1.36
query23	17.08	1.08	0.83
query24	3.00	1.89	1.65
query25	0.43	0.11	0.15
query26	0.39	0.15	0.14
query27	0.04	0.05	0.03
query28	9.18	0.53	0.46
query29	12.59	3.19	3.21
query30	0.24	0.06	0.06
query31	2.87	0.39	0.39
query32	3.24	0.47	0.46
query33	2.96	2.95	3.02
query34	16.98	4.53	4.49
query35	4.59	4.53	4.48
query36	0.66	0.48	0.48
query37	0.08	0.06	0.05
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.22 s
Total hot run time: 30.38 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 26, 2025

BE UT Coverage Report

Increment line coverage 33.33% (2/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 42.38% (11535/27215)
Line Coverage 33.37% (100247/300366)
Region Coverage 32.66% (52196/159819)
Branch Coverage 29.73% (28035/94312)

@morrySnow morrySnow merged commit 6946878 into branch-3.1 Jun 26, 2025
20 of 22 checks passed
@morrySnow morrySnow deleted the auto-pick-51874-branch-3.1 branch June 26, 2025 03:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@Thearas @doris-robot @hello-stephen @morrySnow @dataroaring @bobhan1