[fix](inverted index) Clear inverted index cache from file cache #49685

zzzxl1993 · 2025-03-31T06:06:11Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-03-31T06:06:15Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zzzxl1993 · 2025-03-31T06:06:23Z

run buildall

doris-robot · 2025-03-31T07:12:08Z

TPC-H: Total hot run time: 33892 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 32c15cb51c60443d19ade4a59116b1185a790898, data reload: false

------ Round 1 ----------------------------------
q1	26333	5353	5037	5037
q2	2100	289	170	170
q3	10416	1263	683	683
q4	10237	1004	517	517
q5	7547	2387	2318	2318
q6	181	165	136	136
q7	929	730	613	613
q8	9330	1325	1045	1045
q9	6831	5151	5077	5077
q10	6795	2353	1914	1914
q11	473	280	262	262
q12	344	354	218	218
q13	17784	3632	3126	3126
q14	228	226	208	208
q15	526	493	492	492
q16	631	610	571	571
q17	573	859	340	340
q18	7789	7125	7022	7022
q19	1509	960	567	567
q20	337	334	196	196
q21	3850	3339	2419	2419
q22	1096	1026	961	961
Total cold run time: 115839 ms
Total hot run time: 33892 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5143	5107	5090	5090
q2	247	325	231	231
q3	2142	2662	2268	2268
q4	1409	1815	1422	1422
q5	4511	4464	4347	4347
q6	211	167	125	125
q7	1956	1903	1746	1746
q8	2639	2539	2611	2539
q9	7193	7228	7279	7228
q10	3005	3204	2742	2742
q11	588	508	489	489
q12	705	757	621	621
q13	3484	3895	3277	3277
q14	292	302	274	274
q15	527	473	482	473
q16	645	695	638	638
q17	1155	1550	1396	1396
q18	7833	7435	7394	7394
q19	810	799	820	799
q20	1992	1996	1845	1845
q21	5155	4707	4657	4657
q22	1103	1022	979	979
Total cold run time: 52745 ms
Total hot run time: 50580 ms

doris-robot · 2025-03-31T07:23:41Z

TPC-DS: Total hot run time: 185968 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 32c15cb51c60443d19ade4a59116b1185a790898, data reload: false

query1	1009	479	478	478
query2	6539	1961	1947	1947
query3	6817	221	226	221
query4	26299	23610	23432	23432
query5	4329	654	464	464
query6	292	192	188	188
query7	4611	480	282	282
query8	304	242	241	241
query9	8634	2579	2565	2565
query10	468	311	254	254
query11	15201	15011	14816	14816
query12	150	107	100	100
query13	1636	509	387	387
query14	8715	5997	6074	5997
query15	196	182	177	177
query16	7128	664	452	452
query17	1098	713	556	556
query18	1957	403	297	297
query19	188	180	153	153
query20	118	113	114	113
query21	211	120	98	98
query22	4176	4215	4105	4105
query23	33895	33199	32915	32915
query24	8435	2331	2371	2331
query25	548	457	414	414
query26	1234	271	150	150
query27	2750	486	325	325
query28	4412	2421	2392	2392
query29	783	566	453	453
query30	290	218	196	196
query31	931	866	794	794
query32	74	64	64	64
query33	574	371	338	338
query34	774	836	492	492
query35	778	821	797	797
query36	973	978	874	874
query37	115	99	74	74
query38	4156	4158	4067	4067
query39	1422	1433	1398	1398
query40	247	115	102	102
query41	56	53	53	53
query42	121	101	111	101
query43	512	483	458	458
query44	1258	777	790	777
query45	174	171	165	165
query46	809	1009	629	629
query47	1785	1819	1791	1791
query48	363	399	303	303
query49	786	527	423	423
query50	671	725	403	403
query51	4162	4176	4072	4072
query52	109	103	101	101
query53	219	249	175	175
query54	495	477	419	419
query55	82	84	88	84
query56	299	306	260	260
query57	1148	1178	1096	1096
query58	249	240	240	240
query59	2697	2861	2591	2591
query60	296	275	280	275
query61	140	127	133	127
query62	784	734	657	657
query63	215	185	174	174
query64	4274	1045	696	696
query65	4330	4259	4256	4256
query66	1157	424	366	366
query67	15978	15462	15443	15443
query68	7744	868	509	509
query69	464	310	268	268
query70	1222	1111	1139	1111
query71	455	288	271	271
query72	5374	4754	4853	4754
query73	699	633	344	344
query74	9082	9174	8714	8714
query75	3818	3236	2660	2660
query76	3600	1169	746	746
query77	799	360	284	284
query78	9949	10143	9264	9264
query79	1899	854	571	571
query80	669	530	434	434
query81	471	256	215	215
query82	210	123	96	96
query83	177	173	168	168
query84	263	92	82	82
query85	751	357	315	315
query86	381	303	279	279
query87	4482	4689	4301	4301
query88	3479	2224	2199	2199
query89	376	307	285	285
query90	2035	213	210	210
query91	149	142	111	111
query92	75	66	59	59
query93	1818	1033	585	585
query94	613	417	308	308
query95	361	270	265	265
query96	485	561	273	273
query97	3148	3245	3152	3152
query98	233	218	207	207
query99	1332	1409	1294	1294
Total cold run time: 272249 ms
Total hot run time: 185968 ms

doris-robot · 2025-03-31T07:28:58Z

ClickBench: Total hot run time: 30.74 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 32c15cb51c60443d19ade4a59116b1185a790898, data reload: false

query1	0.04	0.03	0.03
query2	0.12	0.10	0.10
query3	0.25	0.19	0.18
query4	1.58	0.18	0.20
query5	0.58	0.57	0.58
query6	1.21	0.72	0.71
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.56	0.53	0.50
query10	0.57	0.58	0.57
query11	0.15	0.11	0.11
query12	0.14	0.12	0.11
query13	0.61	0.60	0.61
query14	2.67	2.78	2.70
query15	0.91	0.85	0.86
query16	0.38	0.37	0.39
query17	1.03	1.06	1.02
query18	0.22	0.20	0.20
query19	1.89	1.92	1.80
query20	0.01	0.01	0.02
query21	15.36	0.90	0.55
query22	0.75	1.20	0.63
query23	14.97	1.38	0.65
query24	7.26	1.36	0.51
query25	0.52	0.22	0.07
query26	0.62	0.17	0.14
query27	0.05	0.05	0.04
query28	9.59	0.87	0.42
query29	12.69	3.98	3.28
query30	0.25	0.09	0.07
query31	2.82	0.59	0.38
query32	3.23	0.54	0.47
query33	3.01	3.09	3.04
query34	15.83	5.10	4.50
query35	4.53	4.56	4.53
query36	0.65	0.50	0.47
query37	0.09	0.07	0.06
query38	0.04	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.03
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.58 s
Total hot run time: 30.74 s

freemandealer

LGTM

github-actions · 2025-03-31T12:33:44Z

PR approved by anyone and no changes requested.

deardeng

LGTM

github-actions · 2025-03-31T16:32:01Z

PR approved by at least one committer and no changes requested.

zzzxl1993 · 2025-04-01T03:46:51Z

run buildall

doris-robot · 2025-04-01T05:03:37Z

TPC-H: Total hot run time: 34612 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 77503b06b05a443eac40716871a619bc6234f895, data reload: false

------ Round 1 ----------------------------------
q1	26121	5014	5041	5014
q2	2063	288	187	187
q3	10388	1261	726	726
q4	10220	1021	546	546
q5	7514	2376	2374	2374
q6	187	162	134	134
q7	944	760	621	621
q8	9339	1356	1204	1204
q9	6785	5172	5113	5113
q10	6822	2304	1890	1890
q11	473	292	273	273
q12	355	356	220	220
q13	17939	3714	3141	3141
q14	241	232	213	213
q15	536	491	480	480
q16	629	617	595	595
q17	629	882	376	376
q18	7686	7228	7308	7228
q19	1474	970	571	571
q20	334	339	229	229
q21	4205	3453	2563	2563
q22	1090	1007	914	914
Total cold run time: 115974 ms
Total hot run time: 34612 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5214	5138	5149	5138
q2	248	329	240	240
q3	2214	2643	2305	2305
q4	1438	1927	1470	1470
q5	4529	4421	4396	4396
q6	218	163	128	128
q7	1992	1920	1748	1748
q8	2668	2647	2590	2590
q9	7296	7118	7145	7118
q10	3028	3166	2747	2747
q11	585	507	482	482
q12	717	780	658	658
q13	3526	3960	3380	3380
q14	273	309	270	270
q15	523	498	489	489
q16	653	695	654	654
q17	1174	1563	1388	1388
q18	7907	7702	7541	7541
q19	847	827	887	827
q20	1910	1968	1850	1850
q21	5388	4970	4953	4953
q22	1036	1066	1001	1001
Total cold run time: 53384 ms
Total hot run time: 51373 ms

doris-robot · 2025-04-01T05:15:33Z

TPC-DS: Total hot run time: 193023 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 77503b06b05a443eac40716871a619bc6234f895, data reload: false

query1	1416	1064	1056	1056
query2	6242	1994	1973	1973
query3	11055	4632	4643	4632
query4	26054	23774	22888	22888
query5	4757	601	454	454
query6	318	220	208	208
query7	3992	521	285	285
query8	305	262	253	253
query9	8524	2590	2574	2574
query10	462	334	260	260
query11	15556	15110	14909	14909
query12	169	120	106	106
query13	1555	519	388	388
query14	9211	6309	6193	6193
query15	213	190	167	167
query16	7282	654	461	461
query17	1161	796	619	619
query18	2003	441	340	340
query19	210	210	171	171
query20	131	133	122	122
query21	210	132	115	115
query22	4602	4656	4154	4154
query23	34312	33586	33374	33374
query24	8609	2385	2433	2385
query25	525	475	414	414
query26	1201	274	157	157
query27	3136	508	340	340
query28	4801	2480	2444	2444
query29	728	637	448	448
query30	285	247	199	199
query31	917	869	787	787
query32	76	62	64	62
query33	537	365	312	312
query34	798	887	550	550
query35	837	879	790	790
query36	983	1017	894	894
query37	130	100	75	75
query38	4223	4453	4221	4221
query39	1477	1494	1422	1422
query40	229	129	114	114
query41	54	55	54	54
query42	125	104	120	104
query43	532	528	485	485
query44	1347	833	815	815
query45	183	178	168	168
query46	849	1018	655	655
query47	1837	1858	1789	1789
query48	394	443	318	318
query49	786	546	462	462
query50	662	694	409	409
query51	4258	4256	4261	4256
query52	111	108	106	106
query53	232	268	191	191
query54	608	609	512	512
query55	85	80	91	80
query56	310	323	300	300
query57	1187	1222	1133	1133
query58	266	278	276	276
query59	2780	2938	2727	2727
query60	351	341	306	306
query61	138	147	135	135
query62	763	738	673	673
query63	235	195	198	195
query64	4058	1085	759	759
query65	4541	4368	4365	4365
query66	1179	441	313	313
query67	16176	15550	15340	15340
query68	8222	905	515	515
query69	516	324	280	280
query70	1240	1147	1121	1121
query71	459	327	300	300
query72	5608	4841	4926	4841
query73	715	669	357	357
query74	8946	8825	9370	8825
query75	3921	3246	2699	2699
query76	3755	1198	750	750
query77	800	377	295	295
query78	9936	10224	9281	9281
query79	1996	872	571	571
query80	654	527	451	451
query81	480	258	225	225
query82	431	131	100	100
query83	270	255	231	231
query84	260	111	91	91
query85	871	365	326	326
query86	332	315	275	275
query87	4495	4439	4277	4277
query88	3285	2222	2240	2222
query89	408	323	284	284
query90	1959	216	213	213
query91	147	150	115	115
query92	77	65	61	61
query93	1077	953	597	597
query94	669	407	282	282
query95	378	298	292	292
query96	490	566	274	274
query97	3291	3203	3140	3140
query98	228	215	193	193
query99	1449	1355	1297	1297
Total cold run time: 280644 ms
Total hot run time: 193023 ms

doris-robot · 2025-04-01T05:21:00Z

ClickBench: Total hot run time: 31.02 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 77503b06b05a443eac40716871a619bc6234f895, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.10
query3	0.26	0.20	0.20
query4	1.60	0.19	0.19
query5	0.61	0.58	0.61
query6	1.21	0.72	0.72
query7	0.03	0.02	0.02
query8	0.04	0.04	0.04
query9	0.60	0.52	0.55
query10	0.57	0.59	0.58
query11	0.17	0.10	0.11
query12	0.15	0.11	0.12
query13	0.60	0.61	0.59
query14	2.66	2.81	2.73
query15	0.92	0.84	0.85
query16	0.39	0.38	0.39
query17	1.01	1.07	1.06
query18	0.21	0.20	0.19
query19	1.93	2.08	1.81
query20	0.01	0.01	0.02
query21	15.35	0.93	0.55
query22	0.74	1.15	0.62
query23	15.11	1.42	0.60
query24	6.98	1.56	0.72
query25	0.46	0.25	0.09
query26	0.49	0.17	0.13
query27	0.05	0.05	0.05
query28	9.84	0.88	0.43
query29	12.52	4.00	3.29
query30	0.25	0.09	0.06
query31	2.81	0.59	0.39
query32	3.23	0.55	0.48
query33	3.02	3.00	3.10
query34	15.66	5.07	4.52
query35	4.51	4.54	4.49
query36	0.67	0.50	0.48
query37	0.09	0.06	0.07
query38	0.06	0.04	0.03
query39	0.03	0.03	0.03
query40	0.17	0.15	0.13
query41	0.09	0.04	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.34 s
Total hot run time: 31.02 s

doris-robot · 2025-04-01T07:03:47Z

BE UT Coverage Report

Increment line coverage 81.82% (27/33) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	51.35% (13742/26763)
Line Coverage	40.72% (119704/293969)
Region Coverage	39.43% (60968/154614)
Branch Coverage	34.23% (30602/89402)

)

…e cache #49685 (#49738) Cherry-picked from #49685 Co-authored-by: zzzxl <yangsiyu@selectdb.com>

freemandealer · 2025-05-26T15:17:45Z

@zzzxl1993

compaction 的时候是不是也会更新 inverted index，如果 compaction 失败了，inverted index 看着没有被正确清理。FYI:

// be/src/olap/compaction.cpp
void CloudCompactionMixin::garbage_collection() {
    if (!config::enable_file_cache) {
        return;
    }
    if (_output_rs_writer) {
        auto* beta_rowset_writer = dynamic_cast<BaseBetaRowsetWriter*>(_output_rs_writer.get());
        DCHECK(beta_rowset_writer);
        for (const auto& [_, file_writer] : beta_rowset_writer->get_file_writers()) {
            auto file_key = io::BlockFileCache::hash(file_writer->path().filename().native());
            auto* file_cache = io::FileCacheFactory::instance()->get_by_path(file_key);
            file_cache->remove_if_cached_async(file_key);
        }
        // TODO: also clear inverted index file here
    }
}

…che#49685)

dataroaring added usercase Important user case type label dev/3.0.x labels Mar 31, 2025

[fix](inverted index) Clear inverted index cache from file cache

77503b0

zzzxl1993 force-pushed the 2025030311404 branch from 32c15cb to 77503b0 Compare March 31, 2025 12:25

freemandealer approved these changes Mar 31, 2025

View reviewed changes

github-actions bot added the reviewed label Mar 31, 2025

deardeng approved these changes Mar 31, 2025

View reviewed changes

gavinchou approved these changes Mar 31, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 31, 2025

gavinchou added the p0_b label Apr 1, 2025

gavinchou merged commit 53aa5f8 into apache:master Apr 1, 2025
27 of 29 checks passed

github-actions bot pushed a commit that referenced this pull request Apr 1, 2025

[fix](inverted index) Clear inverted index cache from file cache (#49685

3d7b362

)

github-actions bot mentioned this pull request Apr 1, 2025

branch-3.0: [fix](inverted index) Clear inverted index cache from file cache #49685 #49738

Merged

dataroaring pushed a commit that referenced this pull request Apr 2, 2025

branch-3.0: [fix](inverted index) Clear inverted index cache from fil…

ca9ebec

…e cache #49685 (#49738) Cherry-picked from #49685 Co-authored-by: zzzxl <yangsiyu@selectdb.com>

dataroaring added dev/3.0.5-merged and removed dev/3.0.x labels Apr 2, 2025

koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025

[fix](inverted index) Clear inverted index cache from file cache (apa…

f559910

…che#49685)

[fix](inverted index) Clear inverted index cache from file cache #49685

[fix](inverted index) Clear inverted index cache from file cache #49685

Uh oh!

Conversation

zzzxl1993 commented Mar 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Mar 31, 2025

Uh oh!

zzzxl1993 commented Mar 31, 2025

Uh oh!

doris-robot commented Mar 31, 2025

Uh oh!

doris-robot commented Mar 31, 2025

Uh oh!

doris-robot commented Mar 31, 2025

Uh oh!

freemandealer left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 31, 2025

Uh oh!

deardeng left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 31, 2025

Uh oh!

zzzxl1993 commented Apr 1, 2025

Uh oh!

doris-robot commented Apr 1, 2025

Uh oh!

doris-robot commented Apr 1, 2025

Uh oh!

doris-robot commented Apr 1, 2025

Uh oh!

doris-robot commented Apr 1, 2025

BE UT Coverage Report

Uh oh!

Uh oh!

freemandealer commented May 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

zzzxl1993 commented Mar 31, 2025 •

edited

Loading