Skip to content

branch-3.1: [optimization](filecache) speed up filecache warm up #51776 #52626

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jul 2, 2025
Merged

branch-3.1: [optimization](filecache) speed up filecache warm up #51776 #52626

morrySnow merged 1 commit into from
Jul 2, 2025

Conversation

@freemandealer
Copy link
Contributor

@freemandealer freemandealer commented Jul 2, 2025 •
edited by morrySnow
Loading

Cherry-picked from #51776

@freemandealer
[optimization](filecache) speed up filecache warm up
5f6bc00 
pick apache#51776 pick apache#51776 pick apache#51776

this pr does the following:

make file cache downloader worker pool thread num configurable
make warm up job split batch size configurable
split large file downloading task to smaller ones to maintain load balance between threads, thus improve concurrency
use meta info to deduce size of inverted idx file size to reduce S3 HEAD ops
some log print optimization
in our test, this opt can improve more than 3x file cache warm up performance

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
@freemandealer freemandealer requested a review from morrySnow as a code owner July 2, 2025 02:08
@Thearas
Copy link
Contributor

Thearas commented Jul 2, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@freemandealer
Copy link
Contributor Author

freemandealer commented Jul 2, 2025

run buildall

@morrySnow morrySnow changed the title [optimization](filecache) speed up filecache warm up branch-3.1: [optimization](filecache) speed up filecache warm up #51776 Jul 2, 2025
@doris-robot
Copy link

doris-robot commented Jul 2, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/40) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.05% (12382/27482)
Line Coverage 36.03% (109596/304159)
Region Coverage 35.11% (56802/161763)
Branch Coverage 32.24% (30810/95578)

@doris-robot
Copy link

doris-robot commented Jul 2, 2025

TPC-H: Total hot run time: 39856 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5f6bc0047823977d60512ddc381e8374374f270f, data reload: false

------ Round 1 ----------------------------------
q1	17931	6924	6633	6633
q2	2078	179	164	164
q3	10667	1150	1208	1150
q4	10412	719	760	719
q5	7768	3129	2841	2841
q6	214	134	134	134
q7	996	629	613	613
q8	9372	1980	1966	1966
q9	6655	6421	6420	6420
q10	7048	2282	2318	2282
q11	465	260	265	260
q12	396	209	205	205
q13	17770	3000	2984	2984
q14	248	207	212	207
q15	509	473	464	464
q16	481	397	368	368
q17	1011	633	621	621
q18	7488	6601	6795	6601
q19	1322	1016	940	940
q20	476	196	196	196
q21	3997	3260	3110	3110
q22	1097	978	985	978
Total cold run time: 108401 ms
Total hot run time: 39856 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6650	6644	6602	6602
q2	329	226	232	226
q3	2980	2951	2962	2951
q4	2072	1844	1758	1758
q5	5723	5767	5824	5767
q6	211	123	123	123
q7	2269	1852	1820	1820
q8	3343	3548	3537	3537
q9	8738	8878	8847	8847
q10	3555	3548	3524	3524
q11	590	489	494	489
q12	796	599	576	576
q13	10133	3168	3129	3129
q14	322	274	275	274
q15	517	468	471	468
q16	481	425	433	425
q17	1870	1641	1622	1622
q18	8254	7793	7906	7793
q19	1700	1537	1521	1521
q20	2083	1862	1914	1862
q21	5247	4928	5012	4928
q22	1146	1038	1017	1017
Total cold run time: 69009 ms
Total hot run time: 59259 ms

@doris-robot
Copy link

doris-robot commented Jul 2, 2025

TPC-DS: Total hot run time: 196443 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5f6bc0047823977d60512ddc381e8374374f270f, data reload: false

query1	1298	917	907	907
query2	6256	1862	1882	1862
query3	10798	4194	4064	4064
query4	33081	23428	23811	23428
query5	4052	454	448	448
query6	281	176	175	175
query7	3988	309	337	309
query8	282	215	215	215
query9	9640	2611	2604	2604
query10	495	257	280	257
query11	17775	15483	15261	15261
query12	161	100	104	100
query13	1581	459	420	420
query14	9265	6687	6853	6687
query15	240	185	201	185
query16	8059	467	529	467
query17	1640	581	589	581
query18	2149	318	312	312
query19	225	155	160	155
query20	125	115	116	115
query21	203	110	102	102
query22	4585	4478	4384	4384
query23	34923	34589	34048	34048
query24	11618	2961	2858	2858
query25	685	395	436	395
query26	1775	183	183	183
query27	2788	362	358	358
query28	8138	2216	2205	2205
query29	1046	459	462	459
query30	255	163	164	163
query31	1052	810	828	810
query32	97	55	55	55
query33	785	317	321	317
query34	923	514	510	510
query35	932	725	756	725
query36	1121	946	988	946
query37	143	68	71	68
query38	4140	3950	3981	3950
query39	1511	1457	1459	1457
query40	262	108	107	107
query41	53	50	53	50
query42	121	101	104	101
query43	522	475	484	475
query44	1336	828	824	824
query45	186	174	174	174
query46	1198	753	746	746
query47	1983	1908	1916	1908
query48	449	356	369	356
query49	1073	434	409	409
query50	847	437	433	433
query51	7445	7387	7336	7336
query52	105	96	95	95
query53	270	199	188	188
query54	1227	477	523	477
query55	83	76	83	76
query56	264	244	246	244
query57	1332	1179	1209	1179
query58	256	234	207	207
query59	3131	2968	2974	2968
query60	295	263	272	263
query61	115	122	115	115
query62	871	694	700	694
query63	231	186	189	186
query64	5047	667	636	636
query65	3330	3267	3293	3267
query66	1382	305	308	305
query67	15724	15536	15736	15536
query68	4767	596	590	590
query69	443	265	261	261
query70	1198	1107	1051	1051
query71	340	261	266	261
query72	6191	4312	4168	4168
query73	768	356	358	356
query74	10529	8961	9208	8961
query75	3338	2633	2673	2633
query76	2667	1090	1097	1090
query77	391	282	271	271
query78	10467	9498	9513	9498
query79	2451	606	625	606
query80	922	425	424	424
query81	529	228	222	222
query82	674	91	87	87
query83	254	146	151	146
query84	237	83	84	83
query85	1611	310	302	302
query86	473	299	307	299
query87	4325	4170	4255	4170
query88	4199	2423	2400	2400
query89	414	283	291	283
query90	1943	190	189	189
query91	144	108	114	108
query92	63	50	53	50
query93	2433	560	561	560
query94	825	296	303	296
query95	355	258	258	258
query96	621	280	288	280
query97	3307	3129	3211	3129
query98	223	195	197	195
query99	1533	1316	1351	1316
Total cold run time: 305884 ms
Total hot run time: 196443 ms

@doris-robot
Copy link

doris-robot commented Jul 2, 2025

ClickBench: Total hot run time: 29.83 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5f6bc0047823977d60512ddc381e8374374f270f, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.63	0.10	0.11
query5	0.52	0.53	0.51
query6	1.13	0.74	0.73
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.56	0.49	0.48
query10	0.55	0.55	0.56
query11	0.14	0.10	0.11
query12	0.15	0.12	0.10
query13	0.60	0.59	0.60
query14	0.77	0.78	0.80
query15	0.85	0.83	0.84
query16	0.37	0.37	0.38
query17	0.97	1.04	1.00
query18	0.23	0.23	0.21
query19	1.94	1.72	1.90
query20	0.02	0.01	0.01
query21	15.40	0.59	0.60
query22	2.11	2.55	1.60
query23	16.94	0.95	0.79
query24	3.86	1.11	1.03
query25	0.21	0.08	0.08
query26	0.47	0.14	0.14
query27	0.05	0.04	0.05
query28	9.91	0.48	0.51
query29	12.54	3.19	3.18
query30	0.24	0.06	0.05
query31	2.88	0.38	0.38
query32	3.26	0.47	0.46
query33	3.00	2.98	3.05
query34	16.92	4.47	4.46
query35	4.57	4.56	4.55
query36	0.67	0.49	0.47
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.35 s
Total hot run time: 29.83 s

@morrySnow morrySnow merged commit bc65a9c into apache:branch-3.1 Jul 2, 2025
22 of 24 checks passed
etah000 pushed a commit to etah000/doris that referenced this pull request Jul 7, 2025
@freemandealer @etah000
branch-3.1: [optimization](filecache) speed up filecache warm up apac…
7e57545 
…he#51776 (apache#52626)

Cherry-picked from apache#51776

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

+1 more reviewer

@kaijchen kaijchen kaijchen approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@freemandealer @Thearas @doris-robot @kaijchen @morrySnow