Skip to content

Conversation

@swjtu-zhanglei
Copy link
Contributor

@swjtu-zhanglei swjtu-zhanglei commented Jun 9, 2025

  • Add unused rowset state for CloudTablet to recycle file cache data and delete_bitmap
  • the pr is enhancement for the pr https://github.com/apache/doris/pull/50973

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jun 9, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@swjtu-zhanglei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33975 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f790265ebc40a9c49ae4c9afd3a3ed77f998f709, data reload: false

------ Round 1 ----------------------------------
q1	26123	5018	4989	4989
q2	1974	275	180	180
q3	10322	1291	683	683
q4	10239	989	521	521
q5	7731	2373	2375	2373
q6	192	170	129	129
q7	923	758	612	612
q8	9307	1352	1113	1113
q9	6863	5121	5157	5121
q10	6822	2314	1899	1899
q11	481	285	274	274
q12	359	355	213	213
q13	17767	3695	3114	3114
q14	234	231	209	209
q15	566	483	474	474
q16	414	430	372	372
q17	607	861	386	386
q18	7790	7353	7188	7188
q19	1467	962	561	561
q20	327	335	233	233
q21	3710	3201	2383	2383
q22	1016	1029	948	948
Total cold run time: 115234 ms
Total hot run time: 33975 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5078	5002	5034	5002
q2	244	318	219	219
q3	2165	2649	2320	2320
q4	1367	1773	1354	1354
q5	4559	4472	4453	4453
q6	210	168	130	130
q7	2026	1974	1751	1751
q8	2596	2557	2530	2530
q9	7221	7162	7137	7137
q10	2998	3172	2709	2709
q11	565	496	500	496
q12	656	775	641	641
q13	3528	3847	3257	3257
q14	273	285	263	263
q15	533	475	475	475
q16	443	485	444	444
q17	1152	1573	1357	1357
q18	7770	7446	7501	7446
q19	821	841	922	841
q20	1964	2038	1888	1888
q21	4962	4389	4297	4297
q22	1028	1018	986	986
Total cold run time: 52159 ms
Total hot run time: 49996 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186047 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f790265ebc40a9c49ae4c9afd3a3ed77f998f709, data reload: false

query1	1015	477	482	477
query2	6549	1788	1811	1788
query3	6745	225	218	218
query4	26961	23584	23274	23274
query5	4394	631	453	453
query6	294	210	198	198
query7	4623	495	298	298
query8	282	226	223	223
query9	8581	2605	2615	2605
query10	491	340	275	275
query11	15802	14992	14972	14972
query12	164	109	104	104
query13	1657	544	418	418
query14	9389	6206	5997	5997
query15	210	198	165	165
query16	7208	665	482	482
query17	1168	722	571	571
query18	1954	396	300	300
query19	224	181	170	170
query20	123	130	119	119
query21	220	128	111	111
query22	4035	4121	3981	3981
query23	34027	33183	33200	33183
query24	8478	2366	2381	2366
query25	548	464	384	384
query26	1231	262	148	148
query27	2769	495	343	343
query28	4324	2123	2106	2106
query29	755	547	451	451
query30	285	219	185	185
query31	946	863	756	756
query32	80	63	67	63
query33	549	353	323	323
query34	814	849	520	520
query35	793	827	712	712
query36	933	976	886	886
query37	114	98	76	76
query38	4104	4126	4124	4124
query39	1463	1421	1397	1397
query40	201	119	102	102
query41	63	61	61	61
query42	132	113	110	110
query43	513	502	468	468
query44	1312	845	832	832
query45	183	173	173	173
query46	831	1047	637	637
query47	1765	1774	1796	1774
query48	402	429	308	308
query49	729	506	411	411
query50	645	714	400	400
query51	4261	4205	4058	4058
query52	113	110	96	96
query53	229	261	190	190
query54	562	565	495	495
query55	87	82	85	82
query56	318	314	276	276
query57	1140	1157	1076	1076
query58	268	272	250	250
query59	2605	2647	2477	2477
query60	335	311	308	308
query61	122	120	120	120
query62	808	721	660	660
query63	225	188	182	182
query64	4314	991	652	652
query65	4253	4151	4164	4151
query66	1138	413	307	307
query67	15961	15880	15702	15702
query68	6149	895	535	535
query69	472	300	273	273
query70	1164	1125	1118	1118
query71	403	328	305	305
query72	5730	4920	5024	4920
query73	622	588	352	352
query74	8980	9061	8921	8921
query75	3150	3219	2694	2694
query76	3197	1197	749	749
query77	485	467	286	286
query78	9975	10151	9208	9208
query79	2746	826	558	558
query80	649	499	432	432
query81	493	259	225	225
query82	489	125	92	92
query83	286	247	229	229
query84	294	120	94	94
query85	782	343	308	308
query86	384	316	303	303
query87	4349	4444	4380	4380
query88	3599	2278	2264	2264
query89	371	315	277	277
query90	1907	207	204	204
query91	142	145	110	110
query92	77	60	62	60
query93	2071	954	601	601
query94	716	423	301	301
query95	374	288	281	281
query96	495	572	280	280
query97	2766	2760	2622	2622
query98	239	228	204	204
query99	1466	1407	1281	1281
Total cold run time: 272839 ms
Total hot run time: 186047 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.8 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f790265ebc40a9c49ae4c9afd3a3ed77f998f709, data reload: false

query1	0.04	0.03	0.04
query2	0.14	0.10	0.10
query3	0.28	0.19	0.19
query4	1.59	0.20	0.19
query5	0.45	0.43	0.42
query6	1.15	0.66	0.66
query7	0.02	0.02	0.02
query8	0.05	0.03	0.03
query9	0.58	0.53	0.52
query10	0.57	0.60	0.58
query11	0.16	0.11	0.10
query12	0.15	0.11	0.12
query13	0.62	0.60	0.60
query14	0.78	0.81	0.82
query15	0.88	0.86	0.88
query16	0.39	0.38	0.39
query17	1.04	1.08	1.08
query18	0.23	0.21	0.21
query19	2.00	1.82	1.80
query20	0.02	0.01	0.02
query21	15.40	0.88	0.55
query22	0.77	1.10	0.60
query23	15.12	1.39	0.62
query24	7.43	1.10	0.44
query25	0.44	0.25	0.17
query26	0.68	0.17	0.15
query27	0.05	0.05	0.05
query28	9.09	0.89	0.44
query29	12.57	3.93	3.26
query30	0.26	0.10	0.06
query31	2.81	0.60	0.40
query32	3.24	0.54	0.47
query33	3.01	3.10	3.10
query34	15.79	5.09	4.48
query35	4.48	4.50	4.50
query36	0.65	0.50	0.49
query37	0.08	0.06	0.07
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 103.41 s
Total hot run time: 28.8 s

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/34) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.79% (20986/26300)
Line Coverage 72.72% (216752/298074)
Region Coverage 70.90% (127686/180105)
Branch Coverage 64.59% (66086/102322)

@swjtu-zhanglei swjtu-zhanglei changed the title draft: Add unused rowset state for CloudTablet [feat](cloud) Add unused rowset state for CloudTablet Jun 10, 2025
@swjtu-zhanglei
Copy link
Contributor Author

run buildall

@swjtu-zhanglei swjtu-zhanglei marked this pull request as ready for review June 10, 2025 06:36
@swjtu-zhanglei
Copy link
Contributor Author

run buildall

@swjtu-zhanglei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34030 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0eb8703ee3b98aa9c1ce496fa250cfa11a6ccec2, data reload: false

------ Round 1 ----------------------------------
q1	26681	5069	5041	5041
q2	1953	283	193	193
q3	10274	1260	739	739
q4	10220	1003	546	546
q5	7529	2389	2328	2328
q6	187	163	132	132
q7	929	743	616	616
q8	9322	1308	1201	1201
q9	6831	5172	5114	5114
q10	6873	2327	1920	1920
q11	483	280	283	280
q12	353	347	220	220
q13	17777	3755	3088	3088
q14	232	230	211	211
q15	559	498	474	474
q16	432	438	380	380
q17	601	863	355	355
q18	7656	7063	7162	7063
q19	1997	974	559	559
q20	336	333	226	226
q21	4057	3249	2392	2392
q22	1058	1048	952	952
Total cold run time: 116340 ms
Total hot run time: 34030 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5253	5056	5081	5056
q2	246	320	220	220
q3	2176	2674	2286	2286
q4	1362	1860	1485	1485
q5	4460	4402	4426	4402
q6	210	168	131	131
q7	2002	1970	1782	1782
q8	2614	2689	2573	2573
q9	7228	7148	7142	7142
q10	3022	3203	2780	2780
q11	605	519	510	510
q12	657	765	630	630
q13	3549	3927	3307	3307
q14	282	289	296	289
q15	527	506	471	471
q16	450	480	440	440
q17	1163	1524	1411	1411
q18	7690	7530	7443	7443
q19	856	892	1017	892
q20	1999	2001	1843	1843
q21	5117	4547	4457	4457
q22	1136	1070	1011	1011
Total cold run time: 52604 ms
Total hot run time: 50561 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193435 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0eb8703ee3b98aa9c1ce496fa250cfa11a6ccec2, data reload: false

query1	1419	1076	1081	1076
query2	6209	1833	1813	1813
query3	11179	4609	4483	4483
query4	54811	25603	23709	23709
query5	5210	530	466	466
query6	400	210	199	199
query7	5136	522	289	289
query8	315	222	200	200
query9	6756	2632	2659	2632
query10	413	327	289	289
query11	15271	15049	14789	14789
query12	153	108	105	105
query13	1144	525	424	424
query14	10136	6332	6345	6332
query15	203	207	189	189
query16	6998	676	520	520
query17	1109	748	609	609
query18	1528	417	320	320
query19	210	206	195	195
query20	130	131	123	123
query21	218	131	109	109
query22	4476	4552	4355	4355
query23	34471	33768	33768	33768
query24	6695	2412	2426	2412
query25	469	469	418	418
query26	730	271	154	154
query27	2384	520	379	379
query28	3050	2155	2152	2152
query29	585	573	441	441
query30	273	237	190	190
query31	839	852	794	794
query32	73	59	59	59
query33	450	356	308	308
query34	803	897	524	524
query35	799	827	761	761
query36	939	1001	898	898
query37	118	97	77	77
query38	4288	4302	4160	4160
query39	1530	1478	1483	1478
query40	209	115	109	109
query41	63	58	59	58
query42	141	111	111	111
query43	519	493	485	485
query44	1374	839	846	839
query45	181	180	192	180
query46	877	1036	670	670
query47	1864	1859	1832	1832
query48	402	437	317	317
query49	659	514	404	404
query50	683	717	422	422
query51	4173	4364	4184	4184
query52	122	121	102	102
query53	233	259	184	184
query54	591	605	519	519
query55	84	83	82	82
query56	313	324	288	288
query57	1221	1243	1161	1161
query58	276	283	263	263
query59	2787	2767	2670	2670
query60	339	319	311	311
query61	131	125	125	125
query62	742	732	676	676
query63	240	192	193	192
query64	1775	1068	728	728
query65	4327	4151	4204	4151
query66	726	397	305	305
query67	16002	15793	15704	15704
query68	6808	897	532	532
query69	548	383	266	266
query70	1175	1092	1127	1092
query71	500	335	290	290
query72	5951	4702	4652	4652
query73	1126	588	351	351
query74	8893	9204	8740	8740
query75	3745	3220	2689	2689
query76	4286	1191	768	768
query77	610	373	293	293
query78	10220	10122	9314	9314
query79	2626	812	591	591
query80	753	523	475	475
query81	484	261	224	224
query82	430	128	95	95
query83	359	256	251	251
query84	288	108	85	85
query85	853	356	310	310
query86	395	301	277	277
query87	4373	4431	4307	4307
query88	3202	2274	2271	2271
query89	405	311	294	294
query90	1789	215	221	215
query91	150	152	113	113
query92	78	61	57	57
query93	2233	948	572	572
query94	660	420	316	316
query95	385	297	288	288
query96	510	577	275	275
query97	2737	2771	2665	2665
query98	238	208	209	208
query99	1466	1385	1288	1288
Total cold run time: 301250 ms
Total hot run time: 193435 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.92 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0eb8703ee3b98aa9c1ce496fa250cfa11a6ccec2, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.10
query3	0.25	0.19	0.20
query4	1.60	0.20	0.20
query5	0.46	0.44	0.45
query6	1.17	0.68	0.67
query7	0.03	0.02	0.01
query8	0.05	0.03	0.04
query9	0.60	0.54	0.52
query10	0.58	0.57	0.58
query11	0.16	0.11	0.11
query12	0.16	0.12	0.12
query13	0.62	0.60	0.60
query14	0.79	0.81	0.81
query15	0.87	0.86	0.86
query16	0.38	0.37	0.37
query17	1.02	1.04	1.04
query18	0.22	0.21	0.22
query19	1.95	1.88	1.79
query20	0.01	0.02	0.02
query21	15.40	0.88	0.54
query22	0.76	1.29	0.77
query23	14.82	1.37	0.66
query24	7.67	1.34	0.35
query25	0.37	0.19	0.07
query26	0.67	0.17	0.13
query27	0.05	0.05	0.04
query28	8.56	0.90	0.45
query29	12.52	3.98	3.34
query30	0.25	0.09	0.06
query31	2.83	0.61	0.39
query32	3.23	0.55	0.47
query33	3.04	3.08	3.16
query34	15.70	5.11	4.54
query35	4.56	4.52	4.55
query36	0.66	0.50	0.50
query37	0.08	0.06	0.07
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.14
query41	0.07	0.03	0.03
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 102.62 s
Total hot run time: 28.92 s

@dataroaring dataroaring added dev/3.0.x usercase Important user case type label labels Jun 10, 2025
@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 0.00% (0/29) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.39% (15076/26733)
Line Coverage 45.14% (134662/298298)
Region Coverage 44.24% (67756/153150)
Branch Coverage 38.79% (34741/89554)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/29) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.74% (20980/26311)
Line Coverage 72.71% (216780/298148)
Region Coverage 70.91% (127742/180151)
Branch Coverage 64.56% (66091/102372)

dataroaring
dataroaring previously approved these changes Jun 10, 2025
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 10, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@swjtu-zhanglei
Copy link
Contributor Author

run cloud_p0

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/36) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.74% (20974/26303)
Line Coverage 72.66% (216721/298274)
Region Coverage 70.91% (127742/180150)
Branch Coverage 64.58% (66137/102406)

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 17, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit 914e1a5 into apache:master Jun 17, 2025
22 of 24 checks passed
dataroaring pushed a commit that referenced this pull request Jun 18, 2025
swjtu-zhanglei added a commit to swjtu-zhanglei/incubator-doris that referenced this pull request Jun 20, 2025
morrySnow pushed a commit that referenced this pull request Jun 23, 2025
swjtu-zhanglei added a commit to swjtu-zhanglei/incubator-doris that referenced this pull request Jun 24, 2025
swjtu-zhanglei added a commit to swjtu-zhanglei/incubator-doris that referenced this pull request Jun 25, 2025
morrySnow pushed a commit that referenced this pull request Jun 26, 2025
Hastyshell pushed a commit to Hastyshell/doris that referenced this pull request Jul 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. cloud dev/3.0.7-merged dev/3.1.0-merged reviewed usercase Important user case type label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants