Skip to content

branch-3.1: [opt](load) avoid over-flushing active memtables in memory limiter #52906 #53910

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jul 28, 2025
Merged

branch-3.1: [opt](load) avoid over-flushing active memtables in memory limiter #52906 #53910

morrySnow merged 1 commit into from
Jul 28, 2025

Conversation

@kaijchen
Copy link
Member

@kaijchen kaijchen commented Jul 25, 2025

backport #52906

@kaijchen
[opt](load) avoid over-flushing active memtables in memory limiter (a…
c395dbc 
…pache#52906)

Related PR: apache#41018

Problem Summary:

Update the `_need_flush()` function to subtract both `_queue_mem_usage`
and `_flush_mem_usage` when deciding how much memory needs to be
flushed.

Previously, we only subtracted `_queue_mem_usage`, which could lead to
flushing more active memtables than necessary.

This change ensures that only the required amount of active memtable
memory is flushed, avoiding premature flushes.
This helps prevent creating small segments, which can hurt performance
and storage efficiency.
@kaijchen kaijchen requested a review from morrySnow as a code owner July 25, 2025 09:08
@kaijchen
Copy link
Member Author

kaijchen commented Jul 25, 2025

run buildall

@Thearas
Copy link
Contributor

Thearas commented Jul 25, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@doris-robot
Copy link

doris-robot commented Jul 25, 2025

TPC-H: Total hot run time: 32442 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c395dbc4647e63c69844ebcd19e9d115e56e0985, data reload: false

------ Round 1 ----------------------------------
q1	17578	5434	5352	5352
q2	2058	297	163	163
q3	10565	1252	782	782
q4	10250	861	451	451
q5	8425	2423	2113	2113
q6	186	161	133	133
q7	900	738	609	609
q8	9321	1433	1194	1194
q9	5150	4961	4861	4861
q10	6778	2239	1833	1833
q11	475	282	261	261
q12	328	352	222	222
q13	17770	3657	3000	3000
q14	231	223	206	206
q15	512	470	472	470
q16	440	442	384	384
q17	585	851	356	356
q18	7014	6319	6343	6319
q19	1346	958	537	537
q20	320	355	203	203
q21	2862	2174	1991	1991
q22	1067	1107	1002	1002
Total cold run time: 104161 ms
Total hot run time: 32442 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5515	5411	5417	5411
q2	240	336	244	244
q3	2360	2611	2289	2289
q4	1315	1809	1328	1328
q5	4413	4819	5006	4819
q6	168	157	127	127
q7	2070	1952	1854	1854
q8	2675	2784	2666	2666
q9	7328	7210	7266	7210
q10	3029	3299	2834	2834
q11	564	497	493	493
q12	631	769	575	575
q13	3406	3823	3204	3204
q14	291	306	265	265
q15	531	461	475	461
q16	451	486	443	443
q17	1198	1720	1252	1252
q18	7540	7540	7351	7351
q19	774	1068	1095	1068
q20	2009	2040	1899	1899
q21	5285	4879	4669	4669
q22	1088	1049	1018	1018
Total cold run time: 52881 ms
Total hot run time: 51480 ms

@doris-robot
Copy link

doris-robot commented Jul 25, 2025

TPC-DS: Total hot run time: 197970 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c395dbc4647e63c69844ebcd19e9d115e56e0985, data reload: false

query1	1305	935	894	894
query2	6219	1942	1871	1871
query3	10899	4468	4523	4468
query4	33114	23724	24050	23724
query5	4259	588	437	437
query6	282	183	172	172
query7	3983	495	323	323
query8	281	249	241	241
query9	9547	2599	2557	2557
query10	470	337	251	251
query11	18177	15292	15435	15292
query12	167	107	103	103
query13	1554	533	423	423
query14	9471	7658	7645	7645
query15	242	211	198	198
query16	7923	612	483	483
query17	1609	761	642	642
query18	2108	432	322	322
query19	229	199	166	166
query20	124	115	115	115
query21	210	127	114	114
query22	4545	4484	4458	4458
query23	35059	34370	34049	34049
query24	7658	2736	2733	2733
query25	521	488	418	418
query26	829	294	180	180
query27	2256	494	356	356
query28	4971	2194	2157	2157
query29	616	599	471	471
query30	242	195	174	174
query31	1007	928	865	865
query32	66	62	77	62
query33	507	369	314	314
query34	738	861	521	521
query35	784	834	738	738
query36	1004	1026	984	984
query37	115	94	71	71
query38	4002	4060	4002	4002
query39	1518	1444	1460	1444
query40	206	123	108	108
query41	50	50	49	49
query42	129	109	112	109
query43	515	518	493	493
query44	1332	821	814	814
query45	188	180	180	180
query46	899	1056	688	688
query47	1981	1995	1960	1960
query48	396	433	341	341
query49	723	490	396	396
query50	705	692	433	433
query51	7418	7337	7242	7242
query52	96	104	95	95
query53	235	266	198	198
query54	532	552	487	487
query55	80	76	81	76
query56	278	273	264	264
query57	1263	1269	1225	1225
query58	238	233	219	219
query59	3182	3431	3073	3073
query60	291	279	264	264
query61	118	113	115	113
query62	827	755	715	715
query63	224	191	193	191
query64	3194	1038	634	634
query65	3387	3302	3280	3280
query66	793	416	313	313
query67	16075	15694	15462	15462
query68	7736	832	532	532
query69	502	306	270	270
query70	1180	1065	1089	1065
query71	393	303	266	266
query72	5838	3739	3859	3739
query73	629	742	344	344
query74	10394	9398	9153	9153
query75	3179	3171	2653	2653
query76	3111	1169	754	754
query77	500	361	275	275
query78	10402	10324	9606	9606
query79	3235	834	597	597
query80	757	521	432	432
query81	496	261	224	224
query82	626	116	90	90
query83	155	154	143	143
query84	245	99	75	75
query85	802	373	290	290
query86	393	313	290	290
query87	4291	4366	4253	4253
query88	5073	2416	2400	2400
query89	411	328	293	293
query90	1744	194	191	191
query91	144	137	108	108
query92	66	55	50	50
query93	2353	885	536	536
query94	693	412	305	305
query95	339	278	267	267
query96	495	610	284	284
query97	3186	3271	3115	3115
query98	225	218	197	197
query99	1600	1405	1313	1313
Total cold run time: 295159 ms
Total hot run time: 197970 ms

@doris-robot
Copy link

doris-robot commented Jul 25, 2025

ClickBench: Total hot run time: 28.75 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c395dbc4647e63c69844ebcd19e9d115e56e0985, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.02
query3	0.23	0.07	0.06
query4	1.62	0.10	0.11
query5	0.53	0.53	0.51
query6	1.13	0.72	0.72
query7	0.04	0.01	0.02
query8	0.05	0.03	0.03
query9	0.58	0.49	0.51
query10	0.56	0.55	0.56
query11	0.15	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.60
query14	0.78	0.78	0.79
query15	0.86	0.83	0.83
query16	0.39	0.38	0.40
query17	1.07	1.07	1.06
query18	0.24	0.20	0.22
query19	1.97	1.84	1.92
query20	0.02	0.01	0.01
query21	15.36	0.88	0.58
query22	0.74	0.79	0.69
query23	15.09	1.43	0.65
query24	3.09	0.66	0.84
query25	0.14	0.09	0.20
query26	0.35	0.17	0.14
query27	0.05	0.07	0.08
query28	13.54	1.06	0.44
query29	12.56	3.85	3.22
query30	0.25	0.09	0.07
query31	2.82	0.59	0.38
query32	3.22	0.53	0.45
query33	2.99	3.01	3.04
query34	16.54	5.20	4.56
query35	4.54	4.57	4.54
query36	0.65	0.49	0.47
query37	0.09	0.07	0.06
query38	0.04	0.03	0.03
query39	0.04	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.03	0.02	0.02
Total cold run time: 103.48 s
Total hot run time: 28.75 s

@doris-robot
Copy link

doris-robot commented Jul 25, 2025

BE UT Coverage Report

Increment line coverage 28.57% (2/7) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.26% (12573/27778)
Line Coverage 36.15% (111974/309715)
Region Coverage 35.21% (57891/164394)
Branch Coverage 32.38% (31464/97178)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 25, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.20% (20913/27445)
Line Coverage 69.56% (215261/309481)
Region Coverage 67.61% (128844/190570)
Branch Coverage 61.21% (67110/109630)

1 similar comment
@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 28, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.20% (20913/27445)
Line Coverage 69.56% (215261/309481)
Region Coverage 67.61% (128844/190570)
Branch Coverage 61.21% (67110/109630)

@morrySnow morrySnow changed the title [opt](load) avoid over-flushing active memtables in memory limiter (#52906) branch-3.1: [opt](load) avoid over-flushing active memtables in memory limiter #52906 Jul 28, 2025
@morrySnow morrySnow merged commit 0280243 into apache:branch-3.1 Jul 28, 2025
22 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kaijchen @Thearas @doris-robot @hello-stephen @morrySnow