[optimization](filecache) speed up filecache warm up #51776

freemandealer · 2025-06-17T01:51:24Z

this pr does the following:

make file cache downloader worker pool thread num configurable
make warm up job split batch size configurable
split large file downloading task to smaller ones to maintain load balance between threads, thus improve concurrency
use meta info to deduce size of inverted idx file size to reduce S3 HEAD ops
some log print optimization

in our test, this opt can improve more than 3x file cache warm up performance

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-06-17T01:51:29Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

freemandealer · 2025-06-17T01:54:32Z

run buildall

freemandealer · 2025-06-17T02:19:44Z

run buildall

doris-robot · 2025-06-17T04:05:54Z

TPC-H: Total hot run time: 34199 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c3effe6c4cbddbc343cce0c082d883cf014a8705, data reload: false

------ Round 1 ----------------------------------
q1	17616	5295	5084	5084
q2	1978	311	169	169
q3	10385	1235	757	757
q4	10221	1018	534	534
q5	7731	2431	2281	2281
q6	202	165	136	136
q7	905	779	631	631
q8	9320	1402	1114	1114
q9	6789	5070	5144	5070
q10	6907	2387	1964	1964
q11	484	295	290	290
q12	353	351	228	228
q13	17800	3723	3113	3113
q14	238	230	216	216
q15	585	498	513	498
q16	434	431	381	381
q17	607	882	365	365
q18	7555	7250	7172	7172
q19	1327	980	572	572
q20	343	345	223	223
q21	4100	3288	2421	2421
q22	1079	1011	980	980
Total cold run time: 106959 ms
Total hot run time: 34199 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5125	5106	5091	5091
q2	245	327	224	224
q3	2177	2689	2299	2299
q4	1376	1812	1364	1364
q5	4283	4201	4356	4201
q6	219	175	136	136
q7	2011	1985	1774	1774
q8	2635	2618	2595	2595
q9	7267	7163	7216	7163
q10	3114	3244	2802	2802
q11	617	513	505	505
q12	674	791	628	628
q13	3571	3908	3259	3259
q14	288	293	285	285
q15	524	484	467	467
q16	476	512	471	471
q17	1201	1572	1409	1409
q18	7900	7525	7355	7355
q19	828	778	851	778
q20	2001	2071	1958	1958
q21	5117	4461	4305	4305
q22	1109	1038	1025	1025
Total cold run time: 52758 ms
Total hot run time: 50094 ms

doris-robot · 2025-06-17T04:17:46Z

TPC-DS: Total hot run time: 193271 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c3effe6c4cbddbc343cce0c082d883cf014a8705, data reload: false

query1	1416	1036	1000	1000
query2	6317	1839	1831	1831
query3	11120	4800	4622	4622
query4	25933	23741	23194	23194
query5	4484	621	481	481
query6	312	222	206	206
query7	3991	497	285	285
query8	283	237	217	217
query9	8505	2634	2668	2634
query10	484	319	284	284
query11	15445	15271	14852	14852
query12	168	115	112	112
query13	1558	538	427	427
query14	9261	6218	6268	6218
query15	225	196	168	168
query16	7618	626	499	499
query17	1175	747	604	604
query18	2030	431	375	375
query19	197	185	173	173
query20	129	129	137	129
query21	206	131	115	115
query22	4432	4391	4406	4391
query23	34830	33661	33519	33519
query24	8550	2408	2420	2408
query25	517	468	401	401
query26	764	279	167	167
query27	2828	525	359	359
query28	4620	2198	2220	2198
query29	645	573	456	456
query30	273	235	200	200
query31	912	883	795	795
query32	75	66	64	64
query33	580	398	356	356
query34	827	872	574	574
query35	842	854	758	758
query36	988	1003	900	900
query37	112	103	79	79
query38	4192	4346	4138	4138
query39	1506	1500	1465	1465
query40	223	129	123	123
query41	71	69	70	69
query42	125	113	120	113
query43	538	567	475	475
query44	1336	835	827	827
query45	184	172	169	169
query46	865	1014	678	678
query47	1840	1832	1785	1785
query48	394	436	321	321
query49	695	480	394	394
query50	663	680	407	407
query51	4231	4330	4200	4200
query52	118	118	107	107
query53	243	250	185	185
query54	583	578	534	534
query55	88	85	87	85
query56	349	316	304	304
query57	1215	1289	1194	1194
query58	274	277	273	273
query59	2745	2721	2680	2680
query60	337	316	302	302
query61	131	143	142	142
query62	773	734	701	701
query63	229	198	187	187
query64	3067	1042	691	691
query65	4400	4367	4318	4318
query66	930	411	318	318
query67	16027	15703	15478	15478
query68	8793	901	521	521
query69	510	303	263	263
query70	1253	1123	1116	1116
query71	477	325	292	292
query72	5696	4818	4916	4818
query73	737	640	357	357
query74	8885	9191	8851	8851
query75	3988	3210	2743	2743
query76	3669	1212	750	750
query77	796	377	292	292
query78	10069	10160	9414	9414
query79	2076	820	602	602
query80	662	508	438	438
query81	502	265	228	228
query82	486	132	95	95
query83	293	249	236	236
query84	291	118	92	92
query85	860	357	325	325
query86	328	312	305	305
query87	4469	4464	4327	4327
query88	2864	2278	2255	2255
query89	407	315	295	295
query90	1884	205	207	205
query91	145	138	112	112
query92	75	59	56	56
query93	1161	943	577	577
query94	676	413	309	309
query95	377	291	280	280
query96	567	563	278	278
query97	2762	2768	2647	2647
query98	238	210	206	206
query99	1430	1402	1272	1272
Total cold run time: 278450 ms
Total hot run time: 193271 ms

doris-robot · 2025-06-17T04:23:01Z

ClickBench: Total hot run time: 29.78 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c3effe6c4cbddbc343cce0c082d883cf014a8705, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.04	0.03
query3	0.24	0.07	0.06
query4	1.62	0.11	0.11
query5	0.46	0.41	0.43
query6	1.17	0.67	0.70
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.59	0.53	0.52
query10	0.58	0.57	0.57
query11	0.17	0.11	0.12
query12	0.16	0.12	0.12
query13	0.64	0.62	0.61
query14	0.86	0.82	0.85
query15	0.92	0.90	0.89
query16	0.39	0.39	0.41
query17	1.10	1.11	1.07
query18	0.23	0.21	0.22
query19	1.98	1.83	1.89
query20	0.01	0.01	0.01
query21	15.39	0.91	0.55
query22	0.79	1.26	0.68
query23	14.85	1.40	0.69
query24	7.70	1.05	0.44
query25	0.51	0.18	0.19
query26	0.75	0.17	0.13
query27	0.07	0.06	0.05
query28	8.91	0.94	0.48
query29	12.55	4.04	3.33
query30	0.26	0.09	0.07
query31	2.83	0.64	0.41
query32	3.24	0.61	0.51
query33	3.09	3.12	3.19
query34	16.06	5.56	4.90
query35	4.93	4.88	4.90
query36	0.71	0.52	0.50
query37	0.10	0.08	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.20	0.15	0.15
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 104.46 s
Total hot run time: 29.78 s

hello-stephen · 2025-06-17T06:40:04Z

BE UT Coverage Report

Increment line coverage 0.00% (0/38) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	56.38% (15050/26693)
Line Coverage	45.14% (134586/298137)
Region Coverage	44.26% (67682/152918)
Branch Coverage	38.84% (34729/89414)

hello-stephen · 2025-06-17T06:43:10Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/78) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.82% (20968/26269)
Line Coverage	72.80% (216901/297957)
Region Coverage	71.09% (127891/179896)
Branch Coverage	64.83% (66277/102228)

github-actions · 2025-06-17T09:36:30Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-06-17T09:36:32Z

PR approved by anyone and no changes requested.

freemandealer · 2025-06-19T08:28:45Z

run buildall

doris-robot · 2025-06-19T10:18:14Z

TPC-H: Total hot run time: 35140 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 797633869c0c37e837ed13d31c07eff467822f9c, data reload: false

------ Round 1 ----------------------------------
q1	17641	5302	5042	5042
q2	1953	313	187	187
q3	10339	1345	733	733
q4	10251	1018	537	537
q5	7614	2401	2330	2330
q6	186	173	134	134
q7	904	735	612	612
q8	9319	1308	1209	1209
q9	6917	5093	5129	5093
q10	6916	2389	1998	1998
q11	495	303	288	288
q12	349	362	224	224
q13	17774	3688	3128	3128
q14	236	225	217	217
q15	579	488	488	488
q16	441	450	387	387
q17	659	877	415	415
q18	7721	7217	7197	7197
q19	1923	963	582	582
q20	352	356	238	238
q21	4067	3273	3089	3089
q22	1075	1018	1012	1012
Total cold run time: 107711 ms
Total hot run time: 35140 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5142	5057	5037	5037
q2	263	326	230	230
q3	2200	2701	2273	2273
q4	1387	1822	1406	1406
q5	4244	4321	4401	4321
q6	217	177	128	128
q7	2020	1940	1749	1749
q8	2591	2605	2553	2553
q9	7157	7160	7211	7160
q10	3099	3256	2789	2789
q11	592	520	486	486
q12	712	846	651	651
q13	3531	3874	3357	3357
q14	292	313	274	274
q15	518	484	485	484
q16	466	506	449	449
q17	1184	1603	1347	1347
q18	7515	7107	7110	7107
q19	834	907	1010	907
q20	1937	2019	1825	1825
q21	4923	4474	4333	4333
q22	1080	1058	1020	1020
Total cold run time: 51904 ms
Total hot run time: 49886 ms

doris-robot · 2025-06-19T10:29:55Z

TPC-DS: Total hot run time: 189771 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 797633869c0c37e837ed13d31c07eff467822f9c, data reload: false

query1	993	395	416	395
query2	6537	1915	1895	1895
query3	6745	235	226	226
query4	26629	23346	23323	23323
query5	4354	641	482	482
query6	314	219	210	210
query7	4629	504	304	304
query8	270	236	235	235
query9	8616	2839	2858	2839
query10	473	354	290	290
query11	15349	15067	14866	14866
query12	167	115	111	111
query13	1654	563	455	455
query14	9793	6287	6316	6287
query15	203	205	167	167
query16	7356	652	449	449
query17	1192	730	570	570
query18	1989	413	341	341
query19	198	194	165	165
query20	129	127	128	127
query21	222	126	121	121
query22	4298	4274	4206	4206
query23	34183	33206	33204	33204
query24	7927	2425	2430	2425
query25	547	472	404	404
query26	1238	277	172	172
query27	2626	509	371	371
query28	4303	2342	2325	2325
query29	804	573	490	490
query30	293	233	195	195
query31	960	824	765	765
query32	78	61	69	61
query33	602	362	334	334
query34	832	872	572	572
query35	799	835	748	748
query36	1020	1027	960	960
query37	121	100	83	83
query38	4170	4188	4221	4188
query39	1498	1397	1423	1397
query40	212	122	116	116
query41	66	62	63	62
query42	136	121	117	117
query43	538	543	524	524
query44	1407	891	896	891
query45	216	175	172	172
query46	887	1053	668	668
query47	1778	1802	1765	1765
query48	413	433	331	331
query49	766	521	402	402
query50	680	709	429	429
query51	4152	4256	4122	4122
query52	124	113	110	110
query53	243	279	192	192
query54	622	609	545	545
query55	92	91	88	88
query56	317	314	314	314
query57	1187	1223	1149	1149
query58	277	262	277	262
query59	2755	2774	2594	2594
query60	355	361	342	342
query61	147	139	132	132
query62	808	750	693	693
query63	238	206	205	205
query64	4500	1114	805	805
query65	4306	4188	4239	4188
query66	1138	421	319	319
query67	15915	15749	15866	15749
query68	8120	952	600	600
query69	480	324	291	291
query70	1276	1198	1205	1198
query71	496	350	322	322
query72	5546	4792	4846	4792
query73	740	644	383	383
query74	8968	9329	9074	9074
query75	3816	3224	2748	2748
query76	3648	1263	798	798
query77	790	375	315	315
query78	10066	10326	9392	9392
query79	1647	882	632	632
query80	619	548	472	472
query81	480	270	232	232
query82	609	134	107	107
query83	271	265	247	247
query84	238	116	99	99
query85	794	391	329	329
query86	361	317	301	301
query87	4459	4535	4498	4498
query88	3177	2545	2448	2448
query89	402	328	307	307
query90	1922	226	211	211
query91	150	150	120	120
query92	74	65	61	61
query93	1101	984	647	647
query94	685	425	308	308
query95	412	314	309	309
query96	552	593	300	300
query97	2767	2836	2691	2691
query98	232	233	206	206
query99	1477	1413	1289	1289
Total cold run time: 274761 ms
Total hot run time: 189771 ms

doris-robot · 2025-06-19T10:35:12Z

ClickBench: Total hot run time: 29.6 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 797633869c0c37e837ed13d31c07eff467822f9c, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.04
query3	0.24	0.07	0.06
query4	1.61	0.10	0.10
query5	0.44	0.42	0.41
query6	1.14	0.68	0.68
query7	0.03	0.02	0.02
query8	0.05	0.04	0.03
query9	0.61	0.53	0.54
query10	0.57	0.60	0.58
query11	0.17	0.11	0.11
query12	0.16	0.12	0.12
query13	0.63	0.62	0.61
query14	0.80	0.88	0.81
query15	0.90	0.90	0.89
query16	0.39	0.40	0.40
query17	1.07	1.10	1.07
query18	0.24	0.22	0.23
query19	2.02	1.90	1.87
query20	0.01	0.01	0.01
query21	15.39	0.96	0.57
query22	0.76	1.27	0.64
query23	14.87	1.44	0.62
query24	7.05	1.13	0.41
query25	0.47	0.17	0.17
query26	0.64	0.18	0.14
query27	0.08	0.06	0.05
query28	9.50	0.98	0.47
query29	12.55	4.14	3.45
query30	0.26	0.10	0.06
query31	2.84	0.62	0.41
query32	3.24	0.56	0.48
query33	3.15	3.07	3.12
query34	16.01	5.41	4.80
query35	4.85	4.88	4.83
query36	0.67	0.52	0.48
query37	0.10	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.18	0.15	0.15
query41	0.09	0.03	0.03
query42	0.03	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 104.05 s
Total hot run time: 29.6 s

hello-stephen · 2025-06-19T11:29:16Z

FE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-06-19T13:09:03Z

BE UT Coverage Report

Increment line coverage 0.00% (0/40) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	56.36% (15055/26713)
Line Coverage	45.13% (134686/298458)
Region Coverage	44.27% (67729/152999)
Branch Coverage	38.86% (34758/89450)

hello-stephen · 2025-06-19T13:12:50Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/80) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	60.88% (16004/26286)
Line Coverage	50.33% (150119/298274)
Region Coverage	47.66% (85788/179994)
Branch Coverage	41.22% (42161/102272)

github-actions · 2025-06-19T13:27:23Z

PR approved by at least one committer and no changes requested.

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

dataroaring · 2025-06-26T14:24:00Z

run buildall

doris-robot · 2025-06-26T14:44:19Z

TPC-H: Total hot run time: 33906 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 838c1ef6baba59be341143837e57841642755423, data reload: false

------ Round 1 ----------------------------------
q1	17581	5257	5061	5061
q2	1933	281	183	183
q3	10411	1304	731	731
q4	10251	1052	525	525
q5	8479	2423	2322	2322
q6	207	157	128	128
q7	884	728	597	597
q8	9326	1331	1053	1053
q9	7366	5121	5151	5121
q10	6934	2394	1944	1944
q11	484	335	280	280
q12	345	356	219	219
q13	17786	3675	3119	3119
q14	223	231	201	201
q15	571	472	481	472
q16	428	428	389	389
q17	627	884	381	381
q18	7728	7217	7028	7028
q19	1081	947	542	542
q20	359	346	224	224
q21	4180	3171	2411	2411
q22	1072	1028	975	975
Total cold run time: 108256 ms
Total hot run time: 33906 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5094	5043	5067	5043
q2	238	325	226	226
q3	2218	2675	2312	2312
q4	1402	1767	1353	1353
q5	4280	4525	4458	4458
q6	212	166	123	123
q7	1952	1917	1793	1793
q8	2582	2669	2548	2548
q9	7091	7133	7128	7128
q10	3095	3224	2786	2786
q11	565	488	463	463
q12	657	751	598	598
q13	3482	3792	3248	3248
q14	268	298	280	280
q15	525	484	473	473
q16	437	482	427	427
q17	1149	1562	1363	1363
q18	7729	7713	7410	7410
q19	805	836	1000	836
q20	1956	2092	1947	1947
q21	4961	4632	4463	4463
q22	1098	1065	992	992
Total cold run time: 51796 ms
Total hot run time: 50270 ms

doris-robot · 2025-06-26T14:56:15Z

TPC-DS: Total hot run time: 189082 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 838c1ef6baba59be341143837e57841642755423, data reload: false

query1	1006	390	389	389
query2	5793	1679	1702	1679
query3	6513	209	209	209
query4	26849	23652	23642	23642
query5	3392	580	447	447
query6	292	211	210	210
query7	4558	496	289	289
query8	276	222	221	221
query9	8478	2691	2687	2687
query10	473	368	277	277
query11	15761	14973	15217	14973
query12	151	110	104	104
query13	1627	535	417	417
query14	8832	5664	5680	5664
query15	196	196	172	172
query16	5955	578	486	486
query17	1114	718	595	595
query18	1897	403	319	319
query19	191	195	160	160
query20	122	117	113	113
query21	212	134	111	111
query22	4123	4170	4487	4170
query23	34854	33633	33729	33633
query24	8367	2433	2432	2432
query25	489	457	400	400
query26	1257	263	145	145
query27	2905	524	346	346
query28	4809	2173	2182	2173
query29	705	576	433	433
query30	277	224	196	196
query31	919	872	755	755
query32	87	61	61	61
query33	578	375	299	299
query34	791	863	515	515
query35	819	850	742	742
query36	946	997	907	907
query37	112	105	83	83
query38	4262	4192	4208	4192
query39	1496	1443	1447	1443
query40	212	113	104	104
query41	53	57	49	49
query42	125	106	107	106
query43	492	491	490	490
query44	1352	836	827	827
query45	174	178	165	165
query46	858	1016	651	651
query47	1924	1900	1821	1821
query48	401	455	314	314
query49	699	507	401	401
query50	673	703	398	398
query51	4291	4193	4276	4193
query52	109	108	100	100
query53	225	259	184	184
query54	607	583	512	512
query55	88	84	80	80
query56	299	302	280	280
query57	1257	1247	1161	1161
query58	261	256	262	256
query59	2725	2883	2689	2689
query60	345	317	308	308
query61	123	121	116	116
query62	780	772	672	672
query63	218	185	187	185
query64	4077	1009	682	682
query65	4406	4391	4362	4362
query66	1100	411	306	306
query67	16949	16590	16123	16123
query68	9577	905	548	548
query69	537	307	280	280
query70	1337	1056	1108	1056
query71	478	323	315	315
query72	5833	4990	5019	4990
query73	2592	651	361	361
query74	9352	9483	9007	9007
query75	4334	3269	2771	2771
query76	5683	1172	716	716
query77	936	379	297	297
query78	10360	10507	9629	9629
query79	8654	836	575	575
query80	697	517	471	471
query81	481	268	223	223
query82	269	132	90	90
query83	380	250	235	235
query84	313	105	81	81
query85	858	370	329	329
query86	341	288	293	288
query87	4605	4542	4428	4428
query88	2960	2355	2305	2305
query89	435	328	297	297
query90	2044	213	205	205
query91	135	143	106	106
query92	76	58	53	53
query93	2991	965	602	602
query94	732	413	285	285
query95	375	288	287	287
query96	518	573	285	285
query97	2816	2844	2807	2807
query98	227	219	215	215
query99	1409	1417	1321	1321
Total cold run time: 288642 ms
Total hot run time: 189082 ms

doris-robot · 2025-06-26T15:01:32Z

ClickBench: Total hot run time: 29.05 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 838c1ef6baba59be341143837e57841642755423, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.24	0.07	0.07
query4	1.61	0.10	0.10
query5	0.42	0.44	0.42
query6	1.20	0.66	0.65
query7	0.03	0.01	0.01
query8	0.04	0.03	0.04
query9	0.59	0.51	0.51
query10	0.58	0.58	0.57
query11	0.16	0.11	0.12
query12	0.15	0.11	0.11
query13	0.62	0.60	0.61
query14	0.79	0.81	0.81
query15	0.90	0.87	0.87
query16	0.39	0.37	0.40
query17	1.04	1.05	1.05
query18	0.23	0.22	0.21
query19	1.96	1.94	1.86
query20	0.01	0.01	0.01
query21	15.46	0.90	0.53
query22	0.75	0.99	0.63
query23	15.18	1.36	0.68
query24	7.07	1.91	0.50
query25	0.51	0.08	0.23
query26	0.68	0.17	0.14
query27	0.07	0.05	0.06
query28	8.43	0.90	0.43
query29	12.59	3.91	3.30
query30	0.25	0.09	0.06
query31	2.84	0.60	0.39
query32	3.25	0.54	0.47
query33	3.06	3.16	3.06
query34	15.97	5.33	4.73
query35	4.80	4.84	4.76
query36	0.70	0.52	0.48
query37	0.08	0.06	0.06
query38	0.05	0.05	0.04
query39	0.04	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.18 s
Total hot run time: 29.05 s

hello-stephen · 2025-06-26T15:35:23Z

FE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2025-06-26T16:15:18Z

BE UT Coverage Report

Increment line coverage 0.00% (0/40) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.04% (15367/26942)
Line Coverage	46.12% (139429/302293)
Region Coverage	45.45% (70642/155414)
Branch Coverage	40.22% (37317/92788)

dataroaring

LGTM

pick apache#51776 pick apache#51776 pick apache#51776 this pr does the following: make file cache downloader worker pool thread num configurable make warm up job split batch size configurable split large file downloading task to smaller ones to maintain load balance between threads, thus improve concurrency use meta info to deduce size of inverted idx file size to reduce S3 HEAD ops some log print optimization in our test, this opt can improve more than 3x file cache warm up performance Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

pick #51776 pick #51776 pick #51776 this pr does the following: make file cache downloader worker pool thread num configurable make warm up job split batch size configurable split large file downloading task to smaller ones to maintain load balance between threads, thus improve concurrency use meta info to deduce size of inverted idx file size to reduce S3 HEAD ops some log print optimization in our test, this opt can improve more than 3x file cache warm up performance Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

pick apache#51776 pick apache#51776 pick apache#51776 this pr does the following: make file cache downloader worker pool thread num configurable make warm up job split batch size configurable split large file downloading task to smaller ones to maintain load balance between threads, thus improve concurrency use meta info to deduce size of inverted idx file size to reduce S3 HEAD ops some log print optimization in our test, this opt can improve more than 3x file cache warm up performance Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

… (#52626) Cherry-picked from #51776 Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

…ache#52556) pick apache#51776 pick apache#51776 pick apache#51776 this pr does the following: make file cache downloader worker pool thread num configurable make warm up job split batch size configurable split large file downloading task to smaller ones to maintain load balance between threads, thus improve concurrency use meta info to deduce size of inverted idx file size to reduce S3 HEAD ops some log print optimization in our test, this opt can improve more than 3x file cache warm up performance Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

…he#51776 (apache#52626) Cherry-picked from apache#51776 Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

freemandealer requested review from dataroaring, gavinchou and w41ter as code owners June 17, 2025 01:51

gavinchou previously approved these changes Jun 17, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 17, 2025

github-actions bot added the reviewed label Jun 17, 2025

freemandealer dismissed gavinchou’s stale review via 7976338 June 19, 2025 08:28

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jun 19, 2025

gavinchou previously approved these changes Jun 19, 2025

View reviewed changes

gavinchou added dev/3.0.x cloud labels Jun 19, 2025

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 19, 2025

freemandealer added 4 commits June 26, 2025 22:23

format

d001b8b

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

fix take a long time warning

a242357

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

response to the reviewer

9202cb4

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

minor fix

838c1ef

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

dataroaring force-pushed the warm-up-opt-2 branch from df25e35 to 838c1ef Compare June 26, 2025 14:23

dataroaring approved these changes Jun 27, 2025

View reviewed changes

dataroaring merged commit 9abe0c5 into apache:master Jun 27, 2025
24 of 26 checks passed

github-actions bot added dev/3.0.x-conflict dev/3.1.x-conflict labels Jun 27, 2025

freemandealer mentioned this pull request Jun 30, 2025

[optimization](filecache) speed up filecache warm up #51776 #52556

Merged

16 tasks

dataroaring added dev/3.0.7-merged and removed dev/3.0.x dev/3.0.x-conflict labels Jul 1, 2025

freemandealer mentioned this pull request Jul 2, 2025

branch-3.1: [optimization](filecache) speed up filecache warm up #51776 #52626

Merged

morrySnow pushed a commit that referenced this pull request Jul 2, 2025

branch-3.1: [optimization](filecache) speed up filecache warm up #51776…

bc65a9c

… (#52626) Cherry-picked from #51776 Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

morrySnow added dev/3.1.0-merged and removed dev/3.1.x dev/3.1.x-conflict labels Jul 2, 2025

etah000 pushed a commit to etah000/doris that referenced this pull request Jul 7, 2025

branch-3.1: [optimization](filecache) speed up filecache warm up apac…

7e57545

…he#51776 (apache#52626) Cherry-picked from apache#51776 Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

gavinchou mentioned this pull request Aug 15, 2025

Release Note 3.0.7 #54868

Open

[optimization](filecache) speed up filecache warm up #51776

[optimization](filecache) speed up filecache warm up #51776

Uh oh!

Conversation

freemandealer commented Jun 17, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jun 17, 2025

Uh oh!

freemandealer commented Jun 17, 2025

Uh oh!

freemandealer commented Jun 17, 2025

Uh oh!

doris-robot commented Jun 17, 2025

Uh oh!

doris-robot commented Jun 17, 2025

Uh oh!

doris-robot commented Jun 17, 2025

Uh oh!

hello-stephen commented Jun 17, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 17, 2025

BE Regression && UT Coverage Report

Uh oh!

github-actions bot commented Jun 17, 2025

Uh oh!

github-actions bot commented Jun 17, 2025

Uh oh!

freemandealer commented Jun 19, 2025

Uh oh!

doris-robot commented Jun 19, 2025

Uh oh!

doris-robot commented Jun 19, 2025

Uh oh!

doris-robot commented Jun 19, 2025

Uh oh!

hello-stephen commented Jun 19, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Jun 19, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 19, 2025

BE Regression && UT Coverage Report

Uh oh!

github-actions bot commented Jun 19, 2025

Uh oh!

dataroaring commented Jun 26, 2025

Uh oh!

doris-robot commented Jun 26, 2025

Uh oh!

doris-robot commented Jun 26, 2025

Uh oh!

doris-robot commented Jun 26, 2025

Uh oh!

hello-stephen commented Jun 26, 2025

FE UT Coverage Report

Uh oh!

hello-stephen commented Jun 26, 2025

BE UT Coverage Report

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants