Skip to content

branch-3.0: [Opt](compaction) Prune rows with delete sign=1 in full compaction #51874 #52256

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jun 30, 2025
Merged

branch-3.0: [Opt](compaction) Prune rows with delete sign=1 in full compaction #51874 #52256

dataroaring merged 1 commit into from
Jun 30, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jun 25, 2025

Cherry-picked from #51874

@github-actions github-actions bot requested a review from dataroaring as a code owner June 25, 2025 02:10
@bobhan1 @dataroaring
[Opt](compaction) Prune rows with delete sign=1 in full compaction (#…
a27bfe0 
…51874)

### What problem does this PR solve?

1. Prune rows with delete sign=1 in full compaction
2. also check duplicate key in base compaction when
`enable_prune_delete_sign_when_base_compaction=false`
@dataroaring dataroaring force-pushed the auto-pick-51874-branch-3.0 branch from 4aa1450 to a27bfe0 Compare June 26, 2025 02:09
@dataroaring
Copy link
Contributor

dataroaring commented Jun 26, 2025

run buildall

1 similar comment
@bobhan1
Copy link
Contributor

bobhan1 commented Jun 27, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 27, 2025

TPC-H: Total hot run time: 39957 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a27bfe07970ba86e890727eda5d2b72167a081dc, data reload: false

------ Round 1 ----------------------------------
q1	17591	6938	6681	6681
q2	2058	171	169	169
q3	10880	1075	1179	1075
q4	10543	736	709	709
q5	7711	2827	2857	2827
q6	216	132	133	132
q7	972	628	607	607
q8	9370	1910	2090	1910
q9	6644	6426	6424	6424
q10	7036	2292	2315	2292
q11	470	268	263	263
q12	396	224	218	218
q13	17801	2960	2986	2960
q14	240	204	209	204
q15	504	450	458	450
q16	486	380	383	380
q17	969	553	521	521
q18	7381	6659	6725	6659
q19	1395	1071	1073	1071
q20	476	201	204	201
q21	4240	3324	3214	3214
q22	1102	990	1002	990
Total cold run time: 108481 ms
Total hot run time: 39957 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6641	6620	6553	6553
q2	327	237	233	233
q3	2931	2797	2995	2797
q4	2101	1854	1832	1832
q5	5784	5758	5722	5722
q6	213	126	136	126
q7	2230	1805	1790	1790
q8	3398	3577	3536	3536
q9	8958	8868	8951	8868
q10	3594	3510	3510	3510
q11	597	502	494	494
q12	781	593	613	593
q13	7405	3123	3230	3123
q14	295	274	263	263
q15	515	456	467	456
q16	489	451	463	451
q17	1846	1634	1628	1628
q18	8312	7764	7594	7594
q19	1871	1624	1552	1552
q20	2135	1780	1806	1780
q21	5148	5109	5067	5067
q22	1167	1061	1063	1061
Total cold run time: 66738 ms
Total hot run time: 59029 ms

@doris-robot
Copy link

doris-robot commented Jun 27, 2025

TPC-DS: Total hot run time: 196168 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a27bfe07970ba86e890727eda5d2b72167a081dc, data reload: false

query1	1292	921	902	902
query2	6382	1913	1827	1827
query3	10950	4386	4368	4368
query4	61386	29814	23482	23482
query5	5155	459	441	441
query6	399	173	192	173
query7	5477	300	307	300
query8	298	209	215	209
query9	8511	2576	2574	2574
query10	452	270	267	267
query11	17828	15132	15791	15132
query12	164	112	108	108
query13	1444	442	428	428
query14	10438	7274	6721	6721
query15	207	187	179	179
query16	6666	475	499	475
query17	1151	581	569	569
query18	1714	316	308	308
query19	206	153	156	153
query20	111	105	112	105
query21	201	105	104	104
query22	4630	4405	4748	4405
query23	34380	33655	33778	33655
query24	6183	2869	2845	2845
query25	521	416	402	402
query26	653	170	171	170
query27	1830	351	356	351
query28	4022	2200	2188	2188
query29	732	479	451	451
query30	247	162	158	158
query31	1010	814	818	814
query32	68	54	55	54
query33	452	309	309	309
query34	936	496	511	496
query35	886	753	756	753
query36	1065	973	968	968
query37	111	70	69	69
query38	4053	3955	3981	3955
query39	1538	1446	1472	1446
query40	201	102	102	102
query41	50	49	49	49
query42	115	105	100	100
query43	528	491	499	491
query44	1166	815	827	815
query45	187	175	168	168
query46	1173	747	726	726
query47	2063	1976	1926	1926
query48	478	386	383	383
query49	727	416	465	416
query50	830	436	426	426
query51	7406	7324	7383	7324
query52	111	95	96	95
query53	264	191	189	189
query54	575	479	469	469
query55	87	81	82	81
query56	273	308	247	247
query57	1328	1245	1229	1229
query58	248	203	206	203
query59	3224	3047	3040	3040
query60	287	247	247	247
query61	109	112	114	112
query62	789	680	669	669
query63	216	187	189	187
query64	1340	653	626	626
query65	3256	3189	3185	3185
query66	703	291	296	291
query67	16044	15478	15727	15478
query68	4133	588	570	570
query69	434	260	266	260
query70	1185	1084	1160	1084
query71	333	257	249	249
query72	6273	4085	3958	3958
query73	745	349	357	349
query74	10218	9238	9300	9238
query75	3346	2582	2655	2582
query76	1996	1020	1123	1020
query77	497	265	282	265
query78	10645	9657	9637	9637
query79	1342	589	584	584
query80	869	429	425	425
query81	499	225	217	217
query82	1216	87	85	85
query83	243	148	138	138
query84	282	80	80	80
query85	883	329	302	302
query86	336	320	289	289
query87	4389	4227	4236	4227
query88	3725	2393	2360	2360
query89	418	290	293	290
query90	2026	184	178	178
query91	179	147	149	147
query92	68	49	52	49
query93	1885	559	552	552
query94	737	293	288	288
query95	349	254	255	254
query96	623	282	276	276
query97	3287	3149	3136	3136
query98	212	200	203	200
query99	1532	1321	1291	1291
Total cold run time: 312432 ms
Total hot run time: 196168 ms

@doris-robot
Copy link

doris-robot commented Jun 27, 2025

ClickBench: Total hot run time: 30.02 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a27bfe07970ba86e890727eda5d2b72167a081dc, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.53	0.50	0.52
query6	1.13	0.72	0.74
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.56	0.50	0.50
query10	0.54	0.57	0.56
query11	0.15	0.10	0.10
query12	0.14	0.11	0.10
query13	0.59	0.59	0.61
query14	0.78	0.80	0.80
query15	0.87	0.83	0.83
query16	0.37	0.36	0.39
query17	1.04	0.97	1.08
query18	0.23	0.22	0.21
query19	1.85	1.79	1.91
query20	0.02	0.01	0.01
query21	15.39	0.60	0.61
query22	2.12	2.30	1.68
query23	17.14	0.95	0.78
query24	2.65	1.50	0.98
query25	0.23	0.06	0.06
query26	0.46	0.14	0.13
query27	0.05	0.06	0.03
query28	10.46	0.55	0.48
query29	12.59	3.21	3.22
query30	0.24	0.06	0.06
query31	2.88	0.40	0.38
query32	3.23	0.47	0.46
query33	2.97	3.00	3.02
query34	17.06	4.54	4.53
query35	4.57	4.57	4.54
query36	0.66	0.48	0.51
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.15	0.12	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.98 s
Total hot run time: 30.02 s

dataroaring
dataroaring approved these changes Jun 28, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor Author

github-actions bot commented Jun 28, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jun 28, 2025
@github-actions
Copy link
Contributor Author

github-actions bot commented Jun 28, 2025

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit 0fb1f0f into branch-3.0 Jun 30, 2025
25 of 30 checks passed
@github-actions github-actions bot deleted the auto-pick-51874-branch-3.0 branch June 30, 2025 07:51
koarz pushed a commit to koarz/doris that referenced this pull request Jul 3, 2025
@github-actions @bobhan1 @koarz
branch-3.0: [Opt](compaction) Prune rows with delete sign=1 in full c…
48c85ac 
…ompaction apache#51874 (apache#52256)

Cherry-picked from apache#51874

Co-authored-by: bobhan1 <baohan@selectdb.com>
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@dataroaring @bobhan1 @doris-robot