Skip to content

branch-3.1: [fix](warmup) avoid calling recycle_cache after rebalance #53339 #53338

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jul 24, 2025
Merged

branch-3.1: [fix](warmup) avoid calling recycle_cache after rebalance #53339 #53338

morrySnow merged 1 commit into from
Jul 24, 2025

Conversation

@kaijchen
Copy link
Member

@kaijchen kaijchen commented Jul 15, 2025 •
edited
Loading

backport #53339

@kaijchen kaijchen requested a review from morrySnow as a code owner July 15, 2025 15:03
@Thearas
Copy link
Contributor

Thearas commented Jul 15, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaijchen
[fix](warmup) avoid calling recycle_cache after rebalance (apache#4169)
0345f71 
`recycle_cached_data()` should not call `recycle_cache()`.
Because rebalance will also call `recycle_cached_data()`.
For compaction, we should call `recycle_cache()` in
`delete_expired_stale_rowsets()` and `clear_cache()`.
@kaijchen kaijchen force-pushed the pick-warmup-rebalance-3.1 branch from 0e0edb6 to 0345f71 Compare July 15, 2025 15:08
@kaijchen kaijchen changed the title [fix](warmup) avoid calling recycle_cache after rebalance [fix](warmup) avoid calling recycle_cache after rebalance (#53339) Jul 15, 2025
@kaijchen
Copy link
Member Author

kaijchen commented Jul 15, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 15, 2025

TPC-H: Total hot run time: 39920 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0345f71f2da0b963c6c94667c455f51a1505ea94, data reload: false

------ Round 1 ----------------------------------
q1	17584	6734	6617	6617
q2	2062	204	170	170
q3	10504	1177	1229	1177
q4	10203	807	710	710
q5	7741	2909	2836	2836
q6	213	135	136	135
q7	995	610	595	595
q8	9584	2008	1998	1998
q9	7890	6431	6468	6431
q10	7020	2266	2346	2266
q11	454	256	263	256
q12	404	214	220	214
q13	17801	3009	3016	3009
q14	242	221	222	221
q15	502	451	469	451
q16	467	383	367	367
q17	996	604	504	504
q18	7316	6599	6641	6599
q19	1314	1018	1015	1015
q20	476	205	200	200
q21	3909	3166	3150	3150
q22	1105	999	1026	999
Total cold run time: 108782 ms
Total hot run time: 39920 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6577	6710	6579	6579
q2	324	232	241	232
q3	2929	2909	2968	2909
q4	2076	1815	1792	1792
q5	5734	5697	5707	5697
q6	211	127	130	127
q7	2205	1820	1817	1817
q8	3329	3544	3480	3480
q9	8850	8912	8904	8904
q10	3538	3511	3518	3511
q11	575	486	494	486
q12	796	605	598	598
q13	7942	3129	3105	3105
q14	311	276	276	276
q15	523	463	456	456
q16	491	438	427	427
q17	1839	1627	1632	1627
q18	8346	7712	7606	7606
q19	1672	1529	1505	1505
q20	2136	1864	1872	1864
q21	5242	5002	4971	4971
q22	1159	1064	1020	1020
Total cold run time: 66805 ms
Total hot run time: 58989 ms

@doris-robot
Copy link

doris-robot commented Jul 15, 2025

TPC-DS: Total hot run time: 194566 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0345f71f2da0b963c6c94667c455f51a1505ea94, data reload: false

query1	1316	909	899	899
query2	6264	1949	1890	1890
query3	10927	4469	4719	4469
query4	32681	23336	23682	23336
query5	3804	452	446	446
query6	294	195	183	183
query7	4066	323	329	323
query8	295	232	250	232
query9	9628	2590	2596	2590
query10	481	264	256	256
query11	18051	15179	15217	15179
query12	160	106	104	104
query13	1552	422	431	422
query14	9450	6796	7353	6796
query15	259	177	197	177
query16	8058	489	504	489
query17	1603	558	589	558
query18	2147	299	311	299
query19	275	167	161	161
query20	124	113	113	113
query21	201	103	104	103
query22	4779	4389	4496	4389
query23	35081	34107	33757	33757
query24	11548	2882	2935	2882
query25	657	408	385	385
query26	1454	174	162	162
query27	2890	350	365	350
query28	7633	2221	2224	2221
query29	944	447	467	447
query30	265	158	168	158
query31	1042	852	845	845
query32	94	56	57	56
query33	776	303	321	303
query34	1147	504	511	504
query35	875	756	727	727
query36	1107	971	962	962
query37	148	73	69	69
query38	4031	3918	3914	3914
query39	1526	1465	1457	1457
query40	263	106	114	106
query41	54	50	49	49
query42	117	104	105	104
query43	529	485	484	484
query44	1286	814	811	811
query45	188	178	172	172
query46	1187	762	729	729
query47	2037	1917	1960	1917
query48	445	344	342	342
query49	980	446	431	431
query50	832	441	421	421
query51	7383	7242	7208	7208
query52	109	97	96	96
query53	256	186	184	184
query54	1228	481	482	481
query55	82	86	87	86
query56	296	289	267	267
query57	1333	1205	1194	1194
query58	246	247	224	224
query59	3191	2992	2975	2975
query60	297	260	252	252
query61	110	103	170	103
query62	855	683	697	683
query63	229	201	193	193
query64	5170	668	659	659
query65	3400	3274	3268	3268
query66	1445	298	297	297
query67	15846	15518	15874	15518
query68	4738	577	580	577
query69	448	263	265	263
query70	1187	1104	1055	1055
query71	348	270	261	261
query72	6253	2458	3826	2458
query73	768	346	349	346
query74	10543	8947	9193	8947
query75	3397	2655	2669	2655
query76	2700	1027	1047	1027
query77	370	267	263	263
query78	10493	9517	9618	9517
query79	1450	603	597	597
query80	900	429	432	429
query81	542	218	217	217
query82	411	90	85	85
query83	248	140	139	139
query84	238	77	75	75
query85	1531	301	290	290
query86	469	298	275	275
query87	4371	4229	4252	4229
query88	4477	2383	2375	2375
query89	406	288	298	288
query90	1951	189	188	188
query91	144	107	107	107
query92	61	49	53	49
query93	2378	557	556	556
query94	816	288	296	288
query95	355	254	257	254
query96	622	281	286	281
query97	3276	3147	3126	3126
query98	211	205	198	198
query99	1499	1295	1284	1284
Total cold run time: 304628 ms
Total hot run time: 194566 ms

@doris-robot
Copy link

doris-robot commented Jul 15, 2025

ClickBench: Total hot run time: 29.77 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0345f71f2da0b963c6c94667c455f51a1505ea94, data reload: false

query1	0.03	0.04	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.12	0.10
query5	0.51	0.51	0.50
query6	1.13	0.72	0.73
query7	0.02	0.01	0.02
query8	0.03	0.03	0.03
query9	0.55	0.49	0.52
query10	0.55	0.55	0.55
query11	0.15	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.60
query14	0.77	0.79	0.81
query15	0.83	0.83	0.82
query16	0.37	0.37	0.38
query17	1.04	1.02	0.97
query18	0.22	0.22	0.22
query19	1.95	1.84	1.83
query20	0.01	0.01	0.01
query21	15.40	0.60	0.59
query22	2.30	2.23	1.55
query23	17.09	1.03	0.78
query24	3.04	1.59	1.09
query25	0.25	0.17	0.05
query26	0.48	0.13	0.13
query27	0.04	0.05	0.04
query28	9.87	0.52	0.45
query29	12.63	3.14	3.16
query30	0.25	0.06	0.08
query31	2.85	0.37	0.38
query32	3.26	0.46	0.45
query33	3.01	2.99	2.99
query34	17.26	4.45	4.48
query35	4.51	4.56	4.52
query36	0.66	0.47	0.47
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.2 s
Total hot run time: 29.77 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 15, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/27) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.21% (12511/27673)
Line Coverage 36.10% (111093/307772)
Region Coverage 35.23% (57487/163191)
Branch Coverage 32.33% (31214/96554)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 15, 2025

BE Regression && UT Coverage Report

Increment line coverage 70.37% (19/27) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.10% (20806/27342)
Line Coverage 69.55% (213904/307536)
Region Coverage 67.69% (127742/188718)
Branch Coverage 61.19% (66519/108710)

@morrySnow morrySnow changed the title [fix](warmup) avoid calling recycle_cache after rebalance (#53339) branch-3.1: [fix](warmup) avoid calling recycle_cache after rebalance #53339 Jul 22, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 22, 2025

BE Regression && UT Coverage Report

Increment line coverage 70.37% (19/27) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.09% (20804/27342)
Line Coverage 69.53% (213841/307536)
Region Coverage 67.67% (127707/188718)
Branch Coverage 61.17% (66496/108710)

@morrySnow morrySnow merged commit bda6d12 into apache:branch-3.1 Jul 24, 2025
24 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kaijchen @Thearas @doris-robot @hello-stephen @morrySnow