Skip to content

[fix](warmup) avoid calling recycle_cache after rebalance (#53339) #53523

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 2 commits into from
Jul 21, 2025
Merged

[fix](warmup) avoid calling recycle_cache after rebalance (#53339) #53523

dataroaring merged 2 commits into from
Jul 21, 2025

Conversation

@kaijchen
Copy link
Member

@kaijchen kaijchen commented Jul 18, 2025

backport #53339

@kaijchen
[fix](warmup) avoid calling recycle_cache after rebalance (apache#4169)
0d64f31 
`recycle_cached_data()` should not call `recycle_cache()`.
Because rebalance will also call `recycle_cached_data()`.
For compaction, we should call `recycle_cache()` in
`delete_expired_stale_rowsets()` and `clear_cache()`.
@kaijchen kaijchen requested a review from dataroaring as a code owner July 18, 2025 06:32
@Thearas
Copy link
Contributor

Thearas commented Jul 18, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaijchen
Copy link
Member Author

kaijchen commented Jul 18, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 18, 2025

TPC-H: Total hot run time: 39906 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2ebe94a081fb26798345a48408a0996f2b0d6ddc, data reload: false

------ Round 1 ----------------------------------
q1	17721	6937	6641	6641
q2	2088	210	177	177
q3	10605	1136	1178	1136
q4	10392	761	842	761
q5	7731	2868	2878	2868
q6	218	135	133	133
q7	970	606	588	588
q8	9350	1983	2055	1983
q9	6683	6400	6415	6400
q10	6988	2243	2281	2243
q11	466	258	253	253
q12	415	214	209	209
q13	17788	3000	3019	3000
q14	236	199	202	199
q15	505	485	466	466
q16	494	383	393	383
q17	994	558	569	558
q18	7356	6744	6713	6713
q19	1442	963	1017	963
q20	464	194	207	194
q21	3904	3046	3103	3046
q22	1108	992	995	992
Total cold run time: 107918 ms
Total hot run time: 39906 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6654	6569	6600	6569
q2	337	230	223	223
q3	2908	2935	2921	2921
q4	2027	1889	1910	1889
q5	5716	5686	5706	5686
q6	216	131	131	131
q7	2272	1768	1821	1768
q8	3372	3526	3564	3526
q9	8702	8918	8886	8886
q10	3581	3475	3484	3475
q11	593	517	516	516
q12	795	631	589	589
q13	8405	3121	3117	3117
q14	296	265	260	260
q15	508	468	474	468
q16	478	440	445	440
q17	1871	1624	1617	1617
q18	8434	7777	7593	7593
q19	1660	1622	1490	1490
q20	2129	1874	1891	1874
q21	5291	5128	4919	4919
q22	1110	1039	1003	1003
Total cold run time: 67355 ms
Total hot run time: 58960 ms

@doris-robot
Copy link

doris-robot commented Jul 18, 2025

TPC-DS: Total hot run time: 195650 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2ebe94a081fb26798345a48408a0996f2b0d6ddc, data reload: false

query1	1277	919	878	878
query2	6265	2001	1890	1890
query3	10829	4263	4215	4215
query4	32992	24050	23354	23354
query5	4412	453	463	453
query6	266	172	180	172
query7	4002	327	314	314
query8	280	225	218	218
query9	9591	2585	2580	2580
query10	462	267	259	259
query11	18014	15318	15206	15206
query12	164	102	103	102
query13	1553	421	412	412
query14	10297	6743	6771	6743
query15	239	183	192	183
query16	7996	481	467	467
query17	1632	599	600	599
query18	2126	316	332	316
query19	218	171	166	166
query20	124	119	116	116
query21	207	106	105	105
query22	4719	4339	4346	4339
query23	35260	33911	33783	33783
query24	11540	2940	2947	2940
query25	625	387	394	387
query26	1394	172	169	169
query27	2589	341	383	341
query28	7245	2175	2184	2175
query29	855	447	429	429
query30	247	168	172	168
query31	1033	812	816	812
query32	94	53	58	53
query33	763	287	306	287
query34	967	512	521	512
query35	870	763	730	730
query36	1092	948	941	941
query37	120	66	68	66
query38	4049	3928	3946	3928
query39	1508	1607	1484	1484
query40	213	98	99	98
query41	48	51	46	46
query42	113	102	101	101
query43	531	499	473	473
query44	1221	816	797	797
query45	181	173	166	166
query46	1175	709	716	709
query47	2000	1907	1907	1907
query48	510	377	403	377
query49	898	385	400	385
query50	838	427	449	427
query51	7444	7279	7139	7139
query52	100	88	88	88
query53	263	189	187	187
query54	1175	485	470	470
query55	81	77	77	77
query56	282	254	263	254
query57	1316	1206	1195	1195
query58	235	224	217	217
query59	3240	3158	3036	3036
query60	287	251	263	251
query61	113	158	109	109
query62	836	707	722	707
query63	236	213	193	193
query64	4240	787	749	749
query65	3353	3267	3303	3267
query66	795	318	312	312
query67	16214	15438	15406	15406
query68	4379	575	572	572
query69	441	270	270	270
query70	1207	1135	1101	1101
query71	340	262	254	254
query72	6468	4121	4053	4053
query73	750	338	340	338
query74	10158	8945	9282	8945
query75	3385	2650	2662	2650
query76	2627	1104	974	974
query77	434	285	278	278
query78	10624	9576	9470	9470
query79	2533	607	600	600
query80	961	430	425	425
query81	550	221	217	217
query82	790	86	89	86
query83	250	142	142	142
query84	238	77	79	77
query85	1600	313	308	308
query86	469	304	294	294
query87	4340	4256	4159	4159
query88	4161	2375	2377	2375
query89	422	291	295	291
query90	1995	190	183	183
query91	184	149	151	149
query92	60	50	52	50
query93	2553	540	536	536
query94	880	306	279	279
query95	357	251	249	249
query96	618	285	283	283
query97	3315	3161	3110	3110
query98	221	204	207	204
query99	1569	1308	1293	1293
Total cold run time: 304733 ms
Total hot run time: 195650 ms

@doris-robot
Copy link

doris-robot commented Jul 18, 2025

ClickBench: Total hot run time: 30.35 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2ebe94a081fb26798345a48408a0996f2b0d6ddc, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.62	0.10	0.10
query5	0.51	0.53	0.53
query6	1.14	0.73	0.74
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.55	0.50	0.50
query10	0.55	0.56	0.55
query11	0.14	0.11	0.10
query12	0.14	0.11	0.11
query13	0.59	0.60	0.58
query14	0.77	0.79	0.81
query15	0.84	0.84	0.83
query16	0.39	0.39	0.39
query17	1.08	1.08	0.97
query18	0.22	0.22	0.21
query19	1.92	1.80	1.86
query20	0.01	0.01	0.01
query21	15.40	0.58	0.57
query22	2.19	1.90	1.75
query23	16.92	0.93	0.79
query24	3.13	1.27	1.77
query25	0.35	0.18	0.21
query26	0.36	0.13	0.14
query27	0.05	0.04	0.03
query28	9.54	0.52	0.51
query29	12.57	3.26	3.24
query30	0.24	0.06	0.05
query31	2.86	0.38	0.38
query32	3.24	0.46	0.45
query33	2.98	2.97	3.04
query34	17.19	4.50	4.40
query35	4.49	4.53	4.49
query36	0.68	0.48	0.49
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 103.54 s
Total hot run time: 30.35 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 18, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/43) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.84% (11118/26571)
Line Coverage 32.39% (95199/293884)
Region Coverage 31.58% (49156/155660)
Branch Coverage 27.96% (25178/90034)

dataroaring
dataroaring approved these changes Jul 21, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 237adc2 into apache:branch-3.0 Jul 21, 2025
22 of 24 checks passed
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kaijchen @Thearas @doris-robot @hello-stephen @dataroaring