Skip to content

Conversation

@Hastyshell
Copy link
Collaborator

@Hastyshell Hastyshell commented Feb 25, 2025

What problem does this PR solve?

Problem Summary:

File cache will only be clear when clear_cache is called if tablet meta no longer exists in ms.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Hastyshell
Copy link
Collaborator Author

run buildall

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@doris-robot
Copy link

TPC-H: Total hot run time: 31531 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 44cbd06a54a0ad2cacab93dcfac799faeb85108c, data reload: false

------ Round 1 ----------------------------------
q1	17601	5226	5084	5084
q2	2053	308	174	174
q3	10882	1249	739	739
q4	10335	1013	527	527
q5	8347	2335	2365	2335
q6	191	169	132	132
q7	910	743	600	600
q8	9301	1259	1066	1066
q9	4894	4697	4741	4697
q10	6884	2288	1889	1889
q11	476	283	251	251
q12	350	348	223	223
q13	17764	3685	3099	3099
q14	231	230	207	207
q15	519	478	450	450
q16	627	640	588	588
q17	595	861	343	343
q18	6719	6296	6128	6128
q19	1499	952	562	562
q20	317	317	189	189
q21	2827	2188	1947	1947
q22	361	332	301	301
Total cold run time: 103683 ms
Total hot run time: 31531 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5217	5377	5134	5134
q2	243	335	232	232
q3	2165	2687	2334	2334
q4	1459	1830	1353	1353
q5	4249	4107	4135	4107
q6	208	164	126	126
q7	1850	1794	1764	1764
q8	2627	2620	2600	2600
q9	7332	7187	7177	7177
q10	2991	3216	2771	2771
q11	588	533	486	486
q12	735	788	615	615
q13	3472	4011	3334	3334
q14	273	303	272	272
q15	506	487	475	475
q16	667	691	629	629
q17	1140	1566	1368	1368
q18	7517	7299	7416	7299
q19	816	841	906	841
q20	1966	2027	1875	1875
q21	5455	5045	4730	4730
q22	630	573	532	532
Total cold run time: 52106 ms
Total hot run time: 50054 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190803 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 44cbd06a54a0ad2cacab93dcfac799faeb85108c, data reload: false

query1	1289	924	936	924
query2	6251	1822	1880	1822
query3	10967	4547	4494	4494
query4	53499	25391	23120	23120
query5	5126	575	493	493
query6	341	198	182	182
query7	4907	516	298	298
query8	305	238	225	225
query9	5730	2524	2501	2501
query10	429	317	258	258
query11	15057	15121	14850	14850
query12	163	113	111	111
query13	1049	522	386	386
query14	11120	7022	6789	6789
query15	197	202	192	192
query16	7104	661	483	483
query17	1095	729	582	582
query18	1554	426	315	315
query19	213	199	170	170
query20	130	123	120	120
query21	208	136	116	116
query22	4408	4671	4364	4364
query23	33857	33336	33371	33336
query24	5715	2432	2427	2427
query25	449	467	421	421
query26	758	281	162	162
query27	2138	497	339	339
query28	2761	2465	2431	2431
query29	593	569	428	428
query30	222	194	159	159
query31	863	851	819	819
query32	69	65	62	62
query33	468	351	333	333
query34	779	852	520	520
query35	812	843	774	774
query36	940	980	866	866
query37	116	99	76	76
query38	4289	4409	4246	4246
query39	1533	1440	1453	1440
query40	219	114	106	106
query41	51	50	53	50
query42	122	111	106	106
query43	509	503	501	501
query44	1306	813	836	813
query45	177	172	175	172
query46	872	1074	655	655
query47	1854	1854	1775	1775
query48	398	419	310	310
query49	695	486	436	436
query50	726	768	446	446
query51	4310	4357	4246	4246
query52	103	103	100	100
query53	240	275	203	203
query54	482	506	412	412
query55	83	78	86	78
query56	273	275	250	250
query57	1177	1214	1118	1118
query58	253	234	250	234
query59	2721	2969	2745	2745
query60	324	268	271	268
query61	120	133	148	133
query62	723	721	663	663
query63	226	192	194	192
query64	2160	1025	699	699
query65	3295	3284	3195	3195
query66	757	398	349	349
query67	15654	15568	15307	15307
query68	6817	907	509	509
query69	536	300	256	256
query70	1186	1129	1098	1098
query71	498	299	281	281
query72	5928	3594	3756	3594
query73	1439	751	362	362
query74	9133	9154	8910	8910
query75	3882	3139	2718	2718
query76	4168	1177	746	746
query77	643	356	276	276
query78	10057	10080	9341	9341
query79	2366	834	588	588
query80	648	525	448	448
query81	517	274	238	238
query82	665	124	96	96
query83	289	168	150	150
query84	293	92	75	75
query85	764	338	300	300
query86	364	334	292	292
query87	4464	4377	4309	4309
query88	3635	2227	2196	2196
query89	409	321	290	290
query90	1945	200	188	188
query91	139	138	112	112
query92	71	58	57	57
query93	1604	1034	582	582
query94	659	421	283	283
query95	353	262	253	253
query96	490	558	268	268
query97	3392	3446	3322	3322
query98	222	199	197	197
query99	1418	1395	1272	1272
Total cold run time: 296328 ms
Total hot run time: 190803 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.83 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 44cbd06a54a0ad2cacab93dcfac799faeb85108c, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.04	0.03
query3	0.23	0.06	0.07
query4	1.61	0.10	0.10
query5	0.57	0.54	0.57
query6	1.21	0.73	0.72
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.59	0.53	0.52
query10	0.58	0.58	0.56
query11	0.16	0.11	0.11
query12	0.14	0.11	0.11
query13	0.63	0.60	0.60
query14	2.69	2.67	2.71
query15	0.92	0.85	0.86
query16	0.38	0.39	0.38
query17	1.05	1.00	1.02
query18	0.21	0.20	0.20
query19	1.91	1.82	1.98
query20	0.01	0.01	0.01
query21	15.36	0.94	0.56
query22	0.74	1.22	0.75
query23	14.82	1.38	0.67
query24	7.71	1.06	0.80
query25	0.45	0.30	0.10
query26	0.54	0.17	0.14
query27	0.05	0.05	0.05
query28	9.32	0.92	0.43
query29	12.58	3.95	3.28
query30	0.25	0.09	0.07
query31	2.82	0.60	0.37
query32	3.22	0.55	0.47
query33	2.98	3.02	3.03
query34	15.63	5.09	4.49
query35	4.49	4.50	4.50
query36	0.65	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.13	0.13
query41	0.08	0.04	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.02
Total cold run time: 105.15 s
Total hot run time: 30.83 s

@Hastyshell
Copy link
Collaborator Author

run beut

@Hastyshell Hastyshell changed the title [fix] Still sync rowsets when tablet not found in ms to clear file cache [fix](file-cache) Still sync rowsets when tablet not found in ms to clear file cache Feb 25, 2025
@Hastyshell Hastyshell force-pushed the fix-sync-tablet-remove-not-found branch from 44cbd06 to 7d743d7 Compare February 26, 2025 13:19
@Hastyshell
Copy link
Collaborator Author

run buildall

Copy link
Contributor

@deardeng deardeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@Hastyshell Hastyshell changed the title [fix](file-cache) Still sync rowsets when tablet not found in ms to clear file cache [fix](file-cache) Clear file cache when tablet meta not found in ms Feb 26, 2025
@doris-robot
Copy link

TPC-H: Total hot run time: 31772 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7d743d77de0c921f53baaf08e0e2f1784ef7eb5e, data reload: false

------ Round 1 ----------------------------------
q1	17612	5156	5132	5132
q2	2046	297	167	167
q3	10425	1304	705	705
q4	10215	1021	548	548
q5	7521	2384	2352	2352
q6	193	173	139	139
q7	889	750	629	629
q8	9328	1293	1085	1085
q9	5046	4930	4743	4743
q10	6843	2336	1887	1887
q11	497	286	255	255
q12	345	359	217	217
q13	17766	3705	3140	3140
q14	237	235	217	217
q15	523	470	467	467
q16	638	627	582	582
q17	570	855	350	350
q18	6558	6295	6174	6174
q19	1210	950	525	525
q20	312	327	195	195
q21	2720	2248	1961	1961
q22	362	331	302	302
Total cold run time: 101856 ms
Total hot run time: 31772 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5195	5147	5163	5147
q2	237	335	230	230
q3	2135	2660	2313	2313
q4	1438	1870	1386	1386
q5	4238	4114	4162	4114
q6	204	165	125	125
q7	1873	1813	1683	1683
q8	2626	2576	2656	2576
q9	7361	7188	7111	7111
q10	3007	3202	2819	2819
q11	580	507	492	492
q12	710	771	636	636
q13	3491	3939	3275	3275
q14	270	307	275	275
q15	505	484	473	473
q16	678	708	639	639
q17	1131	1572	1354	1354
q18	7549	7409	7350	7350
q19	790	776	881	776
q20	1989	2033	1861	1861
q21	5386	4928	5009	4928
q22	623	611	539	539
Total cold run time: 52016 ms
Total hot run time: 50102 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184486 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7d743d77de0c921f53baaf08e0e2f1784ef7eb5e, data reload: false

query1	997	412	414	412
query2	6522	1902	1897	1897
query3	6806	211	208	208
query4	26334	23320	23374	23320
query5	4393	684	522	522
query6	309	200	209	200
query7	4607	494	293	293
query8	303	235	238	235
query9	8593	2565	2592	2565
query10	499	293	270	270
query11	15793	15165	14924	14924
query12	169	115	109	109
query13	1668	521	387	387
query14	9885	6268	6927	6268
query15	214	195	181	181
query16	7648	658	476	476
query17	1294	729	538	538
query18	2003	395	289	289
query19	196	187	147	147
query20	119	117	115	115
query21	213	125	104	104
query22	4300	4155	4206	4155
query23	34003	33212	33102	33102
query24	7640	2368	2382	2368
query25	509	455	395	395
query26	1226	268	156	156
query27	2093	497	317	317
query28	3916	2414	2387	2387
query29	712	562	422	422
query30	236	186	161	161
query31	958	863	835	835
query32	79	65	68	65
query33	557	352	294	294
query34	798	849	486	486
query35	790	825	753	753
query36	973	995	915	915
query37	116	96	82	82
query38	4145	4193	4289	4193
query39	1439	1396	1398	1396
query40	204	120	106	106
query41	55	56	54	54
query42	119	101	104	101
query43	486	505	478	478
query44	1262	792	757	757
query45	175	167	159	159
query46	855	1036	640	640
query47	1778	1814	1716	1716
query48	368	410	293	293
query49	776	530	418	418
query50	673	733	406	406
query51	4189	4216	4114	4114
query52	111	108	92	92
query53	226	260	216	216
query54	471	477	396	396
query55	82	82	82	82
query56	268	264	252	252
query57	1136	1176	1061	1061
query58	245	245	240	240
query59	2645	2767	2652	2652
query60	285	261	250	250
query61	129	121	120	120
query62	861	736	694	694
query63	236	183	190	183
query64	4224	1002	654	654
query65	3213	3139	3128	3128
query66	1055	408	305	305
query67	15699	15427	15430	15427
query68	6293	860	496	496
query69	463	297	276	276
query70	1203	1106	1141	1106
query71	417	301	288	288
query72	5528	3545	3791	3545
query73	740	713	343	343
query74	9050	9101	8723	8723
query75	3197	3165	2687	2687
query76	3276	1188	742	742
query77	482	353	290	290
query78	9979	10165	9336	9336
query79	1488	825	582	582
query80	590	530	444	444
query81	525	277	249	249
query82	198	128	91	91
query83	173	175	155	155
query84	248	98	71	71
query85	748	358	305	305
query86	376	293	278	278
query87	4405	4529	4495	4495
query88	2838	2239	2236	2236
query89	397	370	281	281
query90	1897	200	196	196
query91	133	139	110	110
query92	77	62	55	55
query93	1223	1046	579	579
query94	670	409	298	298
query95	353	266	266	266
query96	480	567	266	266
query97	3320	3432	3287	3287
query98	219	204	208	204
query99	1345	1404	1270	1270
Total cold run time: 267212 ms
Total hot run time: 184486 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.05 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7d743d77de0c921f53baaf08e0e2f1784ef7eb5e, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.04	0.04
query3	0.24	0.07	0.07
query4	1.60	0.10	0.10
query5	0.56	0.55	0.54
query6	1.18	0.71	0.71
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.57	0.54	0.52
query10	0.57	0.58	0.57
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.62	0.60	0.59
query14	2.66	2.74	2.79
query15	0.93	0.84	0.84
query16	0.38	0.37	0.37
query17	1.01	1.00	1.00
query18	0.22	0.19	0.19
query19	1.93	2.02	1.78
query20	0.01	0.02	0.02
query21	15.36	0.92	0.54
query22	0.76	1.13	0.67
query23	15.01	1.38	0.65
query24	6.62	1.85	1.05
query25	0.53	0.20	0.11
query26	0.61	0.16	0.12
query27	0.06	0.05	0.05
query28	9.69	0.91	0.44
query29	12.57	3.91	3.24
query30	0.24	0.09	0.06
query31	2.82	0.58	0.38
query32	3.23	0.57	0.46
query33	2.95	3.00	3.10
query34	15.72	5.16	4.54
query35	4.56	4.55	4.59
query36	0.66	0.49	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.09	0.03	0.03
query42	0.03	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 104.84 s
Total hot run time: 31.05 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 44.77% (11936/26660)
Line Coverage: 34.22% (99900/291935)
Region Coverage: 33.37% (51120/153206)
Branch Coverage: 28.92% (25674/88772)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7d743d77de0c921f53baaf08e0e2f1784ef7eb5e_7d743d77de0c921f53baaf08e0e2f1784ef7eb5e/report/index.html
Increment Report: http://coverage.selectdb-in.cc/coverage/7d743d77de0c921f53baaf08e0e2f1784ef7eb5e_7d743d77de0c921f53baaf08e0e2f1784ef7eb5e/increment_report/index.html

@Hastyshell
Copy link
Collaborator Author

run cloud_p0

1 similar comment
@Hastyshell
Copy link
Collaborator Author

run cloud_p0

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 27, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@gavinchou gavinchou added p0_c and removed approved Indicates a PR has been approved by one committer. labels Feb 27, 2025
@gavinchou gavinchou merged commit 93e8b0d into apache:master Feb 27, 2025
33 of 35 checks passed
github-actions bot pushed a commit that referenced this pull request Feb 27, 2025
…48318)

File cache will only be clear when clear_cache is called if tablet meta
no longer exists in ms.
seawinde pushed a commit to seawinde/doris that referenced this pull request Feb 28, 2025
…pache#48318)

File cache will only be clear when clear_cache is called if tablet meta
no longer exists in ms.
mymeiyi pushed a commit to mymeiyi/doris that referenced this pull request Mar 4, 2025
…pache#48318)

File cache will only be clear when clear_cache is called if tablet meta
no longer exists in ms.
dataroaring pushed a commit that referenced this pull request Mar 10, 2025
…ound in ms #48318 (#48437)

Cherry-picked from #48318

Co-authored-by: Siyang Tang <tangsiyang@selectdb.com>
@gavinchou gavinchou mentioned this pull request Apr 23, 2025
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
…pache#48318)

File cache will only be clear when clear_cache is called if tablet meta
no longer exists in ms.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants