Skip to content

[fix](cold hot separation) Fix the issue that files on the remote storage are not deleted after triggering cold data compaction. #48109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
gavinchou merged 7 commits into from
Feb 27, 2025
Merged

[fix](cold hot separation) Fix the issue that files on the remote storage are not deleted after triggering cold data compaction. #48109

gavinchou merged 7 commits into from
Feb 27, 2025

Conversation

@yagagagaga
Copy link
Contributor

@yagagagaga yagagagaga commented Feb 19, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Feb 19, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

gavinchou
gavinchou previously approved these changes Feb 19, 2025
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 19, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Feb 19, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Feb 19, 2025

PR approved by anyone and no changes requested.

dataroaring
dataroaring previously approved these changes Feb 20, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring added the usercase Important user case type label label Feb 20, 2025
@yagagagaga yagagagaga dismissed stale reviews from dataroaring and gavinchou via 67f7357 February 20, 2025 02:34
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Feb 20, 2025
@doris-robot
Copy link

doris-robot commented Feb 20, 2025

TPC-H: Total hot run time: 31122 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 67f7357f288d3816fa6992f54211864b6a40bf57, data reload: false

------ Round 1 ----------------------------------
q1	17577	5287	5036	5036
q2	2045	310	168	168
q3	10404	1228	737	737
q4	10211	994	538	538
q5	7546	2447	2266	2266
q6	189	168	135	135
q7	905	747	607	607
q8	9302	1291	1144	1144
q9	4879	4467	4694	4467
q10	6850	2321	1864	1864
q11	497	273	251	251
q12	350	354	216	216
q13	17759	3620	3038	3038
q14	228	233	212	212
q15	511	474	456	456
q16	620	619	581	581
q17	572	856	337	337
q18	6758	6173	6101	6101
q19	1363	934	547	547
q20	308	316	195	195
q21	2718	2133	1925	1925
q22	361	329	301	301
Total cold run time: 101953 ms
Total hot run time: 31122 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5096	5086	5063	5063
q2	237	329	221	221
q3	2104	2695	2287	2287
q4	1407	1813	1340	1340
q5	4221	4287	4233	4233
q6	209	168	126	126
q7	1941	1871	1721	1721
q8	2594	2664	2631	2631
q9	7360	7211	7135	7135
q10	3011	3246	2786	2786
q11	563	519	506	506
q12	719	779	645	645
q13	3412	3899	3221	3221
q14	278	290	281	281
q15	503	466	448	448
q16	632	671	635	635
q17	1107	1592	1354	1354
q18	7481	7347	7416	7347
q19	756	768	848	768
q20	1931	2051	1855	1855
q21	5333	4950	4677	4677
q22	617	610	534	534
Total cold run time: 51512 ms
Total hot run time: 49814 ms

@doris-robot
Copy link

doris-robot commented Feb 20, 2025

TPC-DS: Total hot run time: 189779 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 67f7357f288d3816fa6992f54211864b6a40bf57, data reload: false

query1	1306	961	931	931
query2	6187	1811	1844	1811
query3	10984	4601	4278	4278
query4	53041	24416	23446	23446
query5	5238	559	481	481
query6	357	185	189	185
query7	5051	503	296	296
query8	320	227	222	222
query9	5808	2548	2531	2531
query10	423	308	262	262
query11	15294	15201	14890	14890
query12	164	105	113	105
query13	1092	517	400	400
query14	10117	6792	6858	6792
query15	207	218	180	180
query16	7006	666	443	443
query17	1089	721	585	585
query18	1540	414	322	322
query19	203	204	173	173
query20	140	123	134	123
query21	215	129	114	114
query22	4550	4514	4313	4313
query23	33810	33201	33329	33201
query24	5635	2451	2434	2434
query25	457	470	408	408
query26	735	283	154	154
query27	1841	495	371	371
query28	2819	2494	2475	2475
query29	570	586	429	429
query30	208	188	155	155
query31	910	876	806	806
query32	73	62	60	60
query33	468	347	299	299
query34	764	857	519	519
query35	795	846	739	739
query36	948	1010	922	922
query37	122	99	72	72
query38	4293	4382	4261	4261
query39	1521	1432	1454	1432
query40	200	112	104	104
query41	49	49	48	48
query42	126	108	106	106
query43	493	502	467	467
query44	1327	819	831	819
query45	177	176	163	163
query46	896	1048	658	658
query47	1872	1943	1826	1826
query48	412	429	301	301
query49	704	499	415	415
query50	706	755	430	430
query51	4302	4386	4249	4249
query52	116	103	97	97
query53	233	264	193	193
query54	494	537	423	423
query55	80	78	78	78
query56	264	292	240	240
query57	1180	1203	1090	1090
query58	249	269	235	235
query59	2943	2759	2781	2759
query60	290	272	298	272
query61	114	113	110	110
query62	785	715	699	699
query63	230	194	187	187
query64	1836	1030	722	722
query65	3254	3151	3129	3129
query66	718	404	291	291
query67	15966	15711	15261	15261
query68	5434	778	517	517
query69	536	297	264	264
query70	1207	1100	1125	1100
query71	441	292	265	265
query72	5950	3528	3732	3528
query73	1065	756	356	356
query74	9085	9116	8734	8734
query75	3203	3133	2690	2690
query76	3920	1158	725	725
query77	530	368	275	275
query78	10014	10184	9256	9256
query79	2062	849	608	608
query80	691	522	433	433
query81	514	274	238	238
query82	197	131	94	94
query83	176	167	155	155
query84	291	92	77	77
query85	738	350	297	297
query86	348	304	282	282
query87	4433	4558	4422	4422
query88	3502	2224	2206	2206
query89	410	313	291	291
query90	1778	190	188	188
query91	132	138	117	117
query92	76	59	56	56
query93	2228	980	579	579
query94	665	386	291	291
query95	348	266	260	260
query96	481	557	275	275
query97	2737	2845	2721	2721
query98	229	203	204	203
query99	1309	1390	1284	1284
Total cold run time: 291047 ms
Total hot run time: 189779 ms

@doris-robot
Copy link

doris-robot commented Feb 20, 2025

ClickBench: Total hot run time: 31.55 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 67f7357f288d3816fa6992f54211864b6a40bf57, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.03
query3	0.23	0.06	0.07
query4	1.64	0.10	0.10
query5	0.42	0.42	0.40
query6	1.18	0.64	0.66
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.60	0.51	0.53
query10	0.59	0.59	0.58
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.60	0.60
query14	2.70	2.72	2.70
query15	0.92	0.86	0.84
query16	0.37	0.37	0.39
query17	0.99	1.01	1.02
query18	0.22	0.20	0.19
query19	1.87	1.73	2.02
query20	0.02	0.01	0.01
query21	15.37	0.88	0.53
query22	0.75	1.14	0.77
query23	14.89	1.38	0.63
query24	9.55	1.71	1.74
query25	0.52	0.19	0.23
query26	0.91	0.17	0.15
query27	0.05	0.05	0.05
query28	6.73	0.75	0.44
query29	12.53	3.95	3.33
query30	0.24	0.08	0.06
query31	2.84	0.56	0.38
query32	3.22	0.54	0.47
query33	3.07	2.98	3.11
query34	15.70	5.09	4.51
query35	4.49	4.48	4.49
query36	0.67	0.51	0.50
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.02	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 104.82 s
Total hot run time: 31.55 s

@doris-robot
Copy link

doris-robot commented Feb 20, 2025

TPC-H: Total hot run time: 31456 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8692f0a7b59a7a796f4bac73f99fe0530959c838, data reload: false

------ Round 1 ----------------------------------
q1	17638	5323	5050	5050
q2	2041	292	178	178
q3	10413	1311	709	709
q4	10210	1013	543	543
q5	7546	2480	2279	2279
q6	193	166	132	132
q7	896	758	610	610
q8	9315	1308	1181	1181
q9	4926	4703	4577	4577
q10	6837	2324	1874	1874
q11	495	276	257	257
q12	346	357	221	221
q13	17775	3674	3084	3084
q14	230	225	219	219
q15	513	483	468	468
q16	625	602	583	583
q17	585	853	333	333
q18	7003	6232	6220	6220
q19	1210	948	536	536
q20	298	325	190	190
q21	2781	2111	1906	1906
q22	354	318	306	306
Total cold run time: 102230 ms
Total hot run time: 31456 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5081	5060	5074	5060
q2	229	322	233	233
q3	2138	2647	2290	2290
q4	1399	1819	1326	1326
q5	4230	4124	4170	4124
q6	202	160	125	125
q7	1848	1832	1651	1651
q8	2563	2667	2546	2546
q9	7340	7237	7115	7115
q10	2971	3186	2769	2769
q11	562	498	497	497
q12	716	766	597	597
q13	3560	3873	3218	3218
q14	275	289	290	289
q15	503	455	464	455
q16	640	661	630	630
q17	1114	1582	1323	1323
q18	7526	7472	7344	7344
q19	769	800	929	800
q20	1959	1992	1860	1860
q21	5358	4932	4823	4823
q22	626	582	543	543
Total cold run time: 51609 ms
Total hot run time: 49618 ms

@doris-robot
Copy link

doris-robot commented Feb 20, 2025

TPC-DS: Total hot run time: 183545 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8692f0a7b59a7a796f4bac73f99fe0530959c838, data reload: false

query1	981	373	371	371
query2	6537	1857	1779	1779
query3	6792	221	211	211
query4	25969	24010	23442	23442
query5	4583	670	502	502
query6	316	209	222	209
query7	4616	505	300	300
query8	295	250	231	231
query9	8635	2550	2545	2545
query10	443	324	248	248
query11	15213	16534	14856	14856
query12	150	109	105	105
query13	1675	524	389	389
query14	10103	6876	6320	6320
query15	207	188	174	174
query16	7710	621	469	469
query17	1171	722	555	555
query18	1983	395	305	305
query19	187	176	162	162
query20	124	120	115	115
query21	197	123	106	106
query22	4299	4478	4662	4478
query23	34333	33206	32822	32822
query24	7703	2350	2397	2350
query25	544	455	389	389
query26	1222	272	150	150
query27	2107	490	341	341
query28	3882	2426	2387	2387
query29	700	530	420	420
query30	229	188	151	151
query31	922	859	785	785
query32	82	63	62	62
query33	550	330	290	290
query34	754	849	493	493
query35	782	783	723	723
query36	960	989	848	848
query37	123	97	76	76
query38	4230	4225	4091	4091
query39	1492	1412	1391	1391
query40	209	110	99	99
query41	52	47	49	47
query42	125	105	101	101
query43	486	497	460	460
query44	1298	798	791	791
query45	178	166	159	159
query46	856	1035	647	647
query47	1751	1803	1728	1728
query48	395	416	307	307
query49	777	503	420	420
query50	699	746	419	419
query51	4209	4151	4102	4102
query52	104	101	94	94
query53	229	256	187	187
query54	479	476	403	403
query55	85	78	79	78
query56	248	252	248	248
query57	1132	1135	1060	1060
query58	247	256	241	241
query59	2774	2774	2717	2717
query60	290	275	266	266
query61	120	118	118	118
query62	794	718	655	655
query63	228	185	192	185
query64	4260	1047	644	644
query65	3238	3124	3123	3123
query66	1050	396	295	295
query67	15496	15697	15459	15459
query68	2294	788	572	572
query69	423	298	270	270
query70	1192	1110	1097	1097
query71	325	294	277	277
query72	5558	3619	3834	3619
query73	671	737	345	345
query74	9212	9165	8749	8749
query75	3196	3159	2657	2657
query76	2301	1157	765	765
query77	335	375	280	280
query78	9920	10149	9278	9278
query79	1117	856	596	596
query80	1218	531	445	445
query81	548	279	241	241
query82	387	129	93	93
query83	274	165	155	155
query84	241	118	71	71
query85	773	338	291	291
query86	372	306	289	289
query87	4424	4518	4359	4359
query88	2979	2269	2170	2170
query89	391	304	282	282
query90	1865	197	194	194
query91	136	134	108	108
query92	59	58	53	53
query93	1132	990	571	571
query94	658	413	298	298
query95	353	263	253	253
query96	511	544	265	265
query97	2726	2864	2756	2756
query98	217	212	199	199
query99	1277	1406	1253	1253
Total cold run time: 261695 ms
Total hot run time: 183545 ms

@doris-robot
Copy link

doris-robot commented Feb 20, 2025

ClickBench: Total hot run time: 30.26 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8692f0a7b59a7a796f4bac73f99fe0530959c838, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.04	0.03
query3	0.24	0.07	0.06
query4	1.61	0.09	0.10
query5	0.41	0.44	0.39
query6	1.16	0.67	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.60	0.52	0.55
query10	0.59	0.58	0.58
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.64	0.59	0.60
query14	2.67	2.70	2.72
query15	0.91	0.85	0.83
query16	0.39	0.37	0.39
query17	1.02	1.06	1.05
query18	0.22	0.20	0.19
query19	1.88	1.88	2.00
query20	0.01	0.01	0.01
query21	15.35	0.88	0.57
query22	0.76	1.26	0.75
query23	14.79	1.40	0.65
query24	12.47	1.05	0.35
query25	0.32	0.13	0.08
query26	0.49	0.19	0.13
query27	0.06	0.05	0.05
query28	6.22	0.80	0.44
query29	12.56	3.93	3.26
query30	0.25	0.08	0.06
query31	2.83	0.59	0.38
query32	3.22	0.54	0.48
query33	2.98	3.05	3.07
query34	15.79	5.13	4.50
query35	4.51	4.50	4.48
query36	0.66	0.51	0.49
query37	0.08	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.14	0.13
query41	0.07	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.54 s
Total hot run time: 30.26 s

@doris-robot
Copy link

doris-robot commented Feb 20, 2025

TeamCity be ut coverage result:
Function Coverage: 43.82% (11637/26556)
Line Coverage: 33.74% (97534/289092)
Region Coverage: 32.85% (49947/152064)
Branch Coverage: 28.56% (25101/87902)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8692f0a7b59a7a796f4bac73f99fe0530959c838_8692f0a7b59a7a796f4bac73f99fe0530959c838/report/index.html

@dataroaring
Copy link
Contributor

dataroaring commented Feb 20, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Feb 20, 2025

TPC-H: Total hot run time: 31699 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 10a23b104ec9242e070b0ba59be10509f6640701, data reload: false

------ Round 1 ----------------------------------
q1	17600	5323	5092	5092
q2	2059	317	170	170
q3	10401	1323	724	724
q4	10205	1009	523	523
q5	7510	2508	2330	2330
q6	187	170	133	133
q7	919	774	605	605
q8	9312	1350	1156	1156
q9	4912	4597	4609	4597
q10	6813	2318	1886	1886
q11	480	280	259	259
q12	355	365	226	226
q13	17767	3668	3116	3116
q14	228	242	204	204
q15	501	461	459	459
q16	624	629	564	564
q17	585	858	358	358
q18	6541	6363	6345	6345
q19	1211	940	536	536
q20	318	326	193	193
q21	2742	2203	1921	1921
q22	366	329	302	302
Total cold run time: 101636 ms
Total hot run time: 31699 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5125	5119	5120	5119
q2	242	333	243	243
q3	2168	2704	2330	2330
q4	1378	1809	1362	1362
q5	4233	4171	4136	4136
q6	210	161	125	125
q7	1840	1827	1671	1671
q8	2604	2596	2562	2562
q9	7489	7095	7229	7095
q10	3022	3203	2751	2751
q11	587	510	484	484
q12	678	778	620	620
q13	3422	3889	3251	3251
q14	272	291	275	275
q15	514	486	469	469
q16	641	674	640	640
q17	1127	1654	1274	1274
q18	7629	7452	7364	7364
q19	799	1003	1076	1003
q20	1981	1993	1880	1880
q21	5358	4974	4802	4802
q22	602	572	533	533
Total cold run time: 51921 ms
Total hot run time: 49989 ms

@doris-robot
Copy link

doris-robot commented Feb 20, 2025

TPC-DS: Total hot run time: 183318 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 10a23b104ec9242e070b0ba59be10509f6640701, data reload: false

query1	992	376	366	366
query2	6539	1881	1806	1806
query3	6795	218	214	214
query4	26447	23644	23232	23232
query5	4333	659	483	483
query6	290	208	201	201
query7	4613	505	303	303
query8	292	258	241	241
query9	8606	2538	2535	2535
query10	467	311	240	240
query11	15271	15072	14814	14814
query12	157	106	107	106
query13	1637	507	386	386
query14	9081	6059	6006	6006
query15	205	183	176	176
query16	7131	629	435	435
query17	978	729	540	540
query18	1934	387	292	292
query19	191	178	152	152
query20	131	116	116	116
query21	210	120	101	101
query22	4559	4393	4388	4388
query23	34550	33706	33126	33126
query24	7766	2360	2377	2360
query25	527	455	387	387
query26	1229	269	153	153
query27	2535	483	329	329
query28	4233	2403	2374	2374
query29	758	567	415	415
query30	229	185	148	148
query31	902	847	764	764
query32	72	60	59	59
query33	552	340	316	316
query34	770	857	515	515
query35	785	805	738	738
query36	923	992	890	890
query37	111	96	71	71
query38	4097	4068	4025	4025
query39	1445	1405	1388	1388
query40	205	119	100	100
query41	52	48	50	48
query42	116	103	101	101
query43	505	481	470	470
query44	1265	799	794	794
query45	172	168	164	164
query46	855	1021	646	646
query47	1813	1842	1717	1717
query48	392	414	309	309
query49	776	481	413	413
query50	679	741	419	419
query51	4168	4152	4088	4088
query52	104	109	102	102
query53	235	252	191	191
query54	488	489	424	424
query55	87	78	80	78
query56	275	277	267	267
query57	1114	1129	1094	1094
query58	247	231	240	231
query59	2721	2785	2646	2646
query60	280	282	252	252
query61	117	131	115	115
query62	766	712	664	664
query63	227	193	182	182
query64	4348	984	648	648
query65	3196	3132	3128	3128
query66	1118	395	300	300
query67	15832	15508	15340	15340
query68	2420	789	561	561
query69	424	299	256	256
query70	1171	1122	1090	1090
query71	324	277	250	250
query72	5865	3598	3751	3598
query73	648	754	361	361
query74	8891	9152	8941	8941
query75	3095	3149	2696	2696
query76	2222	1175	741	741
query77	343	361	283	283
query78	10063	10122	9313	9313
query79	1007	895	656	656
query80	1163	535	458	458
query81	541	273	241	241
query82	397	127	100	100
query83	269	178	152	152
query84	274	95	70	70
query85	760	341	314	314
query86	374	283	293	283
query87	4549	4454	4356	4356
query88	2895	2220	2217	2217
query89	376	319	282	282
query90	1829	193	188	188
query91	132	131	116	116
query92	63	58	59	58
query93	1082	991	580	580
query94	670	409	291	291
query95	351	258	253	253
query96	511	529	273	273
query97	2796	2887	2766	2766
query98	218	209	205	205
query99	1285	1409	1247	1247
Total cold run time: 261504 ms
Total hot run time: 183318 ms

@doris-robot
Copy link

doris-robot commented Feb 20, 2025

ClickBench: Total hot run time: 30.38 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 10a23b104ec9242e070b0ba59be10509f6640701, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.06	0.07
query4	1.61	0.10	0.10
query5	0.41	0.41	0.41
query6	1.16	0.66	0.66
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.60	0.53	0.53
query10	0.58	0.57	0.58
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.63	0.60	0.61
query14	2.70	2.71	2.68
query15	0.92	0.84	0.85
query16	0.39	0.39	0.38
query17	1.00	1.03	1.03
query18	0.22	0.20	0.19
query19	1.92	1.76	2.01
query20	0.02	0.01	0.02
query21	15.37	0.90	0.52
query22	0.75	1.19	0.71
query23	14.86	1.38	0.58
query24	7.07	2.52	0.64
query25	0.49	0.21	0.09
query26	0.48	0.18	0.13
query27	0.05	0.05	0.06
query28	9.74	0.79	0.42
query29	12.54	4.01	3.32
query30	0.25	0.09	0.06
query31	2.83	0.59	0.39
query32	3.23	0.54	0.46
query33	3.00	3.00	3.01
query34	15.62	5.12	4.52
query35	4.58	4.54	4.55
query36	0.66	0.49	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.14
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.86 s
Total hot run time: 30.38 s

@doris-robot
Copy link

doris-robot commented Feb 20, 2025

TeamCity be ut coverage result:
Function Coverage: 43.83% (11640/26556)
Line Coverage: 33.75% (97557/289090)
Region Coverage: 32.85% (49950/152062)
Branch Coverage: 28.56% (25104/87898)
Coverage Report: http://coverage.selectdb-in.cc/coverage/10a23b104ec9242e070b0ba59be10509f6640701_10a23b104ec9242e070b0ba59be10509f6640701/report/index.html

@yagagagaga
Copy link
Contributor Author

yagagagaga commented Feb 26, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Feb 26, 2025

TPC-H: Total hot run time: 31554 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4c8637a18240e8e4ecdb693f50c35a7dc7d61756, data reload: false

------ Round 1 ----------------------------------
q1	17585	5224	5022	5022
q2	2050	284	167	167
q3	10435	1257	743	743
q4	10200	1019	520	520
q5	7520	2352	2365	2352
q6	185	170	138	138
q7	907	781	595	595
q8	9307	1208	1115	1115
q9	4896	4897	4725	4725
q10	6853	2318	1879	1879
q11	503	275	260	260
q12	355	353	216	216
q13	17799	3671	3084	3084
q14	235	247	211	211
q15	509	469	460	460
q16	620	618	578	578
q17	570	849	349	349
q18	6559	6266	6240	6240
q19	1734	965	538	538
q20	316	332	192	192
q21	2743	2123	1866	1866
q22	370	321	304	304
Total cold run time: 102251 ms
Total hot run time: 31554 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5121	5124	5066	5066
q2	240	323	230	230
q3	2146	2615	2283	2283
q4	1461	1815	1397	1397
q5	4333	4279	4200	4200
q6	209	182	124	124
q7	1891	1865	1704	1704
q8	2601	2759	2618	2618
q9	7390	7314	7130	7130
q10	3010	3203	2768	2768
q11	578	516	512	512
q12	658	780	651	651
q13	3440	3911	3338	3338
q14	281	296	290	290
q15	512	473	462	462
q16	641	675	640	640
q17	1122	1577	1302	1302
q18	7648	7442	7206	7206
q19	800	764	950	764
q20	1956	2050	1887	1887
q21	5488	4933	4698	4698
q22	637	581	546	546
Total cold run time: 52163 ms
Total hot run time: 49816 ms

@doris-robot
Copy link

doris-robot commented Feb 26, 2025

TPC-DS: Total hot run time: 190473 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4c8637a18240e8e4ecdb693f50c35a7dc7d61756, data reload: false

query1	1308	937	927	927
query2	6130	1926	1930	1926
query3	11014	4407	4374	4374
query4	52526	24867	23039	23039
query5	5154	604	487	487
query6	368	197	196	196
query7	4980	513	293	293
query8	319	245	236	236
query9	5955	2594	2591	2591
query10	434	317	248	248
query11	15037	15771	14798	14798
query12	159	109	111	109
query13	1091	516	387	387
query14	10175	6396	6725	6396
query15	192	199	176	176
query16	6987	665	479	479
query17	1081	722	582	582
query18	1548	412	326	326
query19	199	193	174	174
query20	129	177	118	118
query21	204	124	110	110
query22	4646	4904	4559	4559
query23	34086	33305	33296	33296
query24	6294	2406	2411	2406
query25	476	459	403	403
query26	755	272	155	155
query27	1969	475	332	332
query28	2959	2458	2436	2436
query29	605	561	437	437
query30	212	193	155	155
query31	910	884	843	843
query32	77	63	58	58
query33	477	355	315	315
query34	769	864	514	514
query35	795	843	761	761
query36	913	1021	875	875
query37	120	105	78	78
query38	4176	4336	4367	4336
query39	1480	1412	1417	1412
query40	214	114	101	101
query41	51	56	48	48
query42	124	101	107	101
query43	509	550	482	482
query44	1323	804	797	797
query45	190	169	164	164
query46	891	1060	655	655
query47	1877	1921	1875	1875
query48	395	422	316	316
query49	703	491	454	454
query50	729	767	413	413
query51	4293	4340	4227	4227
query52	116	113	91	91
query53	239	272	186	186
query54	495	507	424	424
query55	83	77	84	77
query56	265	274	261	261
query57	1172	1190	1118	1118
query58	239	249	236	236
query59	2916	2981	2816	2816
query60	301	269	259	259
query61	156	117	121	117
query62	722	736	687	687
query63	228	190	187	187
query64	2271	1041	741	741
query65	3309	3151	3139	3139
query66	766	390	295	295
query67	15654	15830	15409	15409
query68	7582	861	486	486
query69	536	295	261	261
query70	1229	1158	1100	1100
query71	494	303	256	256
query72	5775	3645	3733	3645
query73	1437	734	351	351
query74	9085	8921	8642	8642
query75	4078	3162	2737	2737
query76	4307	1175	741	741
query77	684	363	278	278
query78	10018	10211	9316	9316
query79	2037	882	581	581
query80	688	527	507	507
query81	507	278	245	245
query82	481	125	96	96
query83	285	167	147	147
query84	285	89	73	73
query85	795	349	301	301
query86	333	309	284	284
query87	4381	4501	4399	4399
query88	2968	2206	2182	2182
query89	390	311	283	283
query90	1939	191	189	189
query91	127	144	177	144
query92	73	59	55	55
query93	1218	1048	563	563
query94	688	396	286	286
query95	352	263	257	257
query96	497	554	263	263
query97	3410	3364	3287	3287
query98	213	201	204	201
query99	1441	1375	1298	1298
Total cold run time: 295273 ms
Total hot run time: 190473 ms

@doris-robot
Copy link

doris-robot commented Feb 26, 2025

ClickBench: Total hot run time: 31.32 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4c8637a18240e8e4ecdb693f50c35a7dc7d61756, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.04	0.03
query3	0.23	0.07	0.06
query4	1.62	0.11	0.10
query5	0.55	0.64	0.56
query6	1.18	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.58	0.54	0.52
query10	0.56	0.57	0.57
query11	0.15	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.60
query14	2.65	2.82	2.70
query15	0.92	0.85	0.83
query16	0.37	0.37	0.38
query17	1.05	1.02	1.00
query18	0.21	0.20	0.19
query19	1.85	1.76	2.03
query20	0.01	0.02	0.01
query21	15.35	0.90	0.55
query22	0.76	1.19	0.65
query23	14.94	1.36	0.58
query24	7.10	1.70	1.55
query25	0.51	0.18	0.07
query26	0.49	0.15	0.14
query27	0.05	0.06	0.05
query28	10.75	0.75	0.42
query29	12.53	3.92	3.27
query30	0.25	0.09	0.07
query31	2.81	0.59	0.39
query32	3.22	0.54	0.46
query33	3.09	3.00	3.03
query34	15.80	5.15	4.48
query35	4.46	4.55	4.53
query36	0.67	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.14	0.14
query41	0.09	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.11 s
Total hot run time: 31.32 s

@doris-robot
Copy link

doris-robot commented Feb 26, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 44.76% (11933/26657)
Line Coverage 34.20% (99835/291920)
Region Coverage 33.36% (51111/153199)
Branch Coverage 28.91% (25662/88768)

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 27, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Feb 27, 2025

PR approved by at least one committer and no changes requested.

Hastyshell
Hastyshell approved these changes Feb 27, 2025
View reviewed changes
Copy link
Collaborator

@Hastyshell Hastyshell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gavinchou gavinchou merged commit 3d5575e into apache:master Feb 27, 2025
24 of 26 checks passed
@yagagagaga yagagagaga mentioned this pull request Feb 27, 2025
Merged
yagagagaga added a commit to yagagagaga/doris that referenced this pull request Feb 27, 2025
@yagagagaga
[fix](cold hot separation) Fix the issue that files on the remote sto…
cca182b 
…rage are not deleted after triggering cold data compaction. (apache#48109)

(cherry picked from commit 3d5575e)
@yagagagaga yagagagaga mentioned this pull request Feb 27, 2025
Merged
seawinde pushed a commit to seawinde/doris that referenced this pull request Feb 28, 2025
@yagagagaga @seawinde
[fix](cold hot separation) Fix the issue that files on the remote sto…
d935605 
…rage are not deleted after triggering cold data compaction. (apache#48109)
mymeiyi pushed a commit to mymeiyi/doris that referenced this pull request Mar 4, 2025
@yagagagaga @mymeiyi
[fix](cold hot separation) Fix the issue that files on the remote sto…
9a42e8f 
…rage are not deleted after triggering cold data compaction. (apache#48109)
yiguolei pushed a commit to yagagagaga/doris that referenced this pull request Mar 5, 2025
@yagagagaga @yiguolei
[fix](cold hot separation) Fix the issue that files on the remote sto…
e7c7daf 
…rage are not deleted after triggering cold data compaction. (apache#48109)

(cherry picked from commit 3d5575e)
gavinchou pushed a commit that referenced this pull request Mar 10, 2025
@yagagagaga
branch-3.0: [fix](cold hot separation) Fix the issue that files on th…
9ff8d48 
…e remote storage are not deleted after triggering cold data compaction. (#48109) (#48445)

cherry-pick: #48109
@yagagagaga yagagagaga deleted the cir-14348 branch March 10, 2025 12:33
yiguolei pushed a commit that referenced this pull request Mar 12, 2025
@yagagagaga
branch-2.1: [fix](cold hot separation) Fix the issue that files on th…
ef4101b 
…e remote storage are not deleted after triggering cold data compaction. (#48109) (#48446)

Auto pick #48109 branch 2.1
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@yagagagaga @koarz
[fix](cold hot separation) Fix the issue that files on the remote sto…
191b533 
…rage are not deleted after triggering cold data compaction. (apache#48109)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring left review comments

+1 more reviewer

@Hastyshell Hastyshell Hastyshell approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.9-merged dev/3.0.5-merged p0_b reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@yagagagaga @hello-stephen @doris-robot @dataroaring @gavinchou @Hastyshell @yiguolei