Skip to content

[fix](load) fix memtable memory limiter total mem usage #41245

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yiguolei merged 1 commit into from
Sep 25, 2024
Merged

[fix](load) fix memtable memory limiter total mem usage #41245

yiguolei merged 1 commit into from
Sep 25, 2024

Conversation

@kaijchen
Copy link
Member

@kaijchen kaijchen commented Sep 25, 2024

Proposed changes

Previously, mem_usage = write_mem + flush_mem, because active_mem is included in write_mem.

After #40912, write_mem becomes queue_mem, which no longer includes active_mem.
This PR fixes this problem, by setting mem_usage = active_mem + queue_mem + flush_mem

@doris-robot
Copy link

doris-robot commented Sep 25, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@kaijchen kaijchen force-pushed the fix-memtable-limiter-mem-usage branch from 8ec9b75 to 118063f Compare September 25, 2024 02:39
@kaijchen
Copy link
Member Author

kaijchen commented Sep 25, 2024

run buildall

xinyiZzz
xinyiZzz approved these changes Sep 25, 2024
View reviewed changes
Copy link
Contributor

@xinyiZzz xinyiZzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 25, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Sep 25, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 25, 2024

PR approved by anyone and no changes requested.

liaoxin01
liaoxin01 approved these changes Sep 25, 2024
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link

doris-robot commented Sep 25, 2024

TPC-H: Total hot run time: 40721 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 118063fb489da3ee0e9ac3ce6a0b8eb05edb208c, data reload: false

------ Round 1 ----------------------------------
q1	17592	7397	7235	7235
q2	2009	289	287	287
q3	12197	1048	1210	1048
q4	10585	738	738	738
q5	7765	2911	2794	2794
q6	237	149	145	145
q7	981	604	622	604
q8	9469	1983	1998	1983
q9	7678	6388	6367	6367
q10	6958	2287	2305	2287
q11	443	252	260	252
q12	408	218	213	213
q13	17795	2976	2964	2964
q14	266	217	211	211
q15	574	533	517	517
q16	686	626	606	606
q17	976	578	574	574
q18	7231	6717	6609	6609
q19	1397	998	993	993
q20	580	300	292	292
q21	3980	3023	3061	3023
q22	1102	1007	979	979
Total cold run time: 110909 ms
Total hot run time: 40721 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7236	7209	7206	7206
q2	328	225	235	225
q3	3075	3001	2933	2933
q4	2067	1858	1766	1766
q5	5682	5668	5749	5668
q6	227	149	154	149
q7	2216	1778	1784	1778
q8	3381	3510	3496	3496
q9	8801	9021	8922	8922
q10	3559	3522	3595	3522
q11	583	469	490	469
q12	817	594	625	594
q13	9809	3184	3135	3135
q14	305	293	285	285
q15	587	530	524	524
q16	726	666	676	666
q17	1848	1585	1619	1585
q18	8318	7892	7856	7856
q19	1719	1640	1513	1513
q20	2136	1872	1907	1872
q21	5531	5295	5457	5295
q22	1098	1011	1049	1011
Total cold run time: 70049 ms
Total hot run time: 60470 ms

@doris-robot
Copy link

doris-robot commented Sep 25, 2024

TeamCity be ut coverage result:
Function Coverage: 37.31% (9621/25786)
Line Coverage: 28.72% (79632/277288)
Region Coverage: 28.14% (41162/146252)
Branch Coverage: 24.77% (20973/84656)
Coverage Report: http://coverage.selectdb-in.cc/coverage/118063fb489da3ee0e9ac3ce6a0b8eb05edb208c_118063fb489da3ee0e9ac3ce6a0b8eb05edb208c/report/index.html

@doris-robot
Copy link

doris-robot commented Sep 25, 2024

TPC-DS: Total hot run time: 193079 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 118063fb489da3ee0e9ac3ce6a0b8eb05edb208c, data reload: false

query1	949	378	370	370
query2	6379	2060	2031	2031
query3	8701	194	201	194
query4	33841	23621	23533	23533
query5	3411	459	470	459
query6	274	172	162	162
query7	4208	311	301	301
query8	305	224	223	223
query9	9481	2677	2691	2677
query10	466	300	279	279
query11	18080	15194	15330	15194
query12	164	100	96	96
query13	1542	441	442	441
query14	9888	7386	7438	7386
query15	287	168	180	168
query16	8083	529	487	487
query17	1765	597	596	596
query18	2146	339	330	330
query19	391	159	154	154
query20	125	113	119	113
query21	215	115	110	110
query22	4717	4689	4479	4479
query23	35062	34628	34425	34425
query24	10909	2876	2866	2866
query25	630	418	424	418
query26	1135	168	170	168
query27	2217	311	308	308
query28	7545	2436	2445	2436
query29	834	452	429	429
query30	274	162	155	155
query31	1013	805	808	805
query32	97	53	58	53
query33	766	326	300	300
query34	934	519	519	519
query35	874	729	734	729
query36	1092	945	962	945
query37	157	89	91	89
query38	4029	3983	3969	3969
query39	1503	1442	1425	1425
query40	209	98	98	98
query41	50	50	48	48
query42	117	96	99	96
query43	543	460	496	460
query44	1220	840	809	809
query45	196	167	166	166
query46	1148	721	747	721
query47	1909	1828	1844	1828
query48	476	386	386	386
query49	896	406	402	402
query50	824	430	430	430
query51	7073	6883	6877	6877
query52	99	90	86	86
query53	256	190	186	186
query54	1222	482	498	482
query55	80	82	81	81
query56	303	289	269	269
query57	1233	1104	1125	1104
query58	240	228	226	226
query59	3384	3046	2957	2957
query60	298	273	279	273
query61	108	106	106	106
query62	846	674	686	674
query63	228	187	186	186
query64	3943	669	629	629
query65	3275	3219	3189	3189
query66	767	302	321	302
query67	15943	15644	15569	15569
query68	4905	587	574	574
query69	575	306	304	304
query70	1211	1143	1095	1095
query71	395	315	278	278
query72	7379	4111	4186	4111
query73	773	348	357	348
query74	10342	8986	9041	8986
query75	3752	2662	2682	2662
query76	3538	973	924	924
query77	569	293	279	279
query78	10169	9290	10530	9290
query79	2397	600	642	600
query80	1224	449	456	449
query81	564	241	240	240
query82	386	149	149	149
query83	272	140	142	140
query84	279	83	82	82
query85	1578	291	294	291
query86	439	299	297	297
query87	4459	4362	4350	4350
query88	3683	2462	2445	2445
query89	405	293	292	292
query90	2116	191	193	191
query91	200	148	143	143
query92	66	49	52	49
query93	1103	538	539	538
query94	1106	304	310	304
query95	354	263	261	261
query96	608	291	292	291
query97	3311	3161	3161	3161
query98	219	207	203	203
query99	1629	1316	1290	1290
Total cold run time: 301381 ms
Total hot run time: 193079 ms

@doris-robot
Copy link

doris-robot commented Sep 25, 2024

ClickBench: Total hot run time: 32.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 118063fb489da3ee0e9ac3ce6a0b8eb05edb208c, data reload: false

query1	0.04	0.05	0.04
query2	0.06	0.03	0.03
query3	0.23	0.07	0.06
query4	1.64	0.10	0.10
query5	0.52	0.52	0.50
query6	1.14	0.73	0.73
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.51	0.51
query10	0.55	0.58	0.56
query11	0.14	0.11	0.11
query12	0.14	0.12	0.11
query13	0.60	0.60	0.59
query14	3.03	2.96	2.96
query15	0.89	0.84	0.83
query16	0.39	0.41	0.39
query17	1.05	1.07	1.03
query18	0.20	0.19	0.20
query19	1.98	1.82	2.02
query20	0.01	0.01	0.01
query21	15.35	0.60	0.60
query22	2.53	3.13	1.64
query23	17.44	0.88	0.79
query24	2.58	1.24	0.47
query25	0.20	0.11	0.07
query26	0.39	0.14	0.15
query27	0.04	0.04	0.04
query28	11.18	1.10	1.06
query29	12.58	3.25	3.24
query30	0.25	0.07	0.06
query31	2.88	0.39	0.40
query32	3.25	0.47	0.46
query33	2.98	3.01	3.05
query34	16.80	4.46	4.42
query35	4.52	4.49	4.47
query36	0.69	0.50	0.50
query37	0.08	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.35 s
Total hot run time: 32.27 s

@yiguolei yiguolei merged commit c95eff5 into apache:master Sep 25, 2024
kaijchen added a commit to kaijchen/doris that referenced this pull request Oct 21, 2024
@kaijchen
[fix](load) fix memtable memory limiter total mem usage (apache#41245)
311c9ff 
## Proposed changes

Previously, `mem_usage = write_mem + flush_mem`, because `active_mem` is
included in `write_mem`.

After apache#40912, `write_mem` becomes `queue_mem`, which no longer includes
`active_mem`.
This PR fixes this problem, by setting `mem_usage = active_mem +
queue_mem + flush_mem`
@kaijchen kaijchen mentioned this pull request Oct 21, 2024
Merged
liaoxin01 pushed a commit that referenced this pull request Oct 21, 2024
@kaijchen
[fix](load) memtable memory limiter policy changes followup #40912 (#…
5285cbc 
…41018) (#42144)

backport #41018, #41278, #41245
kaijchen added a commit to kaijchen/doris that referenced this pull request Nov 18, 2024
@kaijchen
[fix](load) fix memtable memory limiter total mem usage (apache#41245)
6b06e10 
## Proposed changes

Previously, `mem_usage = write_mem + flush_mem`, because `active_mem` is
included in `write_mem`.

After apache#40912, `write_mem` becomes `queue_mem`, which no longer includes
`active_mem`.
This PR fixes this problem, by setting `mem_usage = active_mem +
queue_mem + flush_mem`
kaijchen added a commit to kaijchen/doris that referenced this pull request Nov 18, 2024
@kaijchen
[fix](load) fix memtable memory limiter total mem usage (apache#41245)
d75a9da 
## Proposed changes

Previously, `mem_usage = write_mem + flush_mem`, because `active_mem` is
included in `write_mem`.

After apache#40912, `write_mem` becomes `queue_mem`, which no longer includes
`active_mem`.
This PR fixes this problem, by setting `mem_usage = active_mem +
queue_mem + flush_mem`
kaijchen added a commit to kaijchen/doris that referenced this pull request Nov 18, 2024
@kaijchen
[fix](load) fix memtable memory limiter total mem usage (apache#41245)
c90743e 
## Proposed changes

Previously, `mem_usage = write_mem + flush_mem`, because `active_mem` is
included in `write_mem`.

After apache#40912, `write_mem` becomes `queue_mem`, which no longer includes
`active_mem`.
This PR fixes this problem, by setting `mem_usage = active_mem +
queue_mem + flush_mem`
@kaijchen kaijchen mentioned this pull request Nov 18, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xinyiZzz xinyiZzz xinyiZzz approved these changes

@liaoxin01 liaoxin01 liaoxin01 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kaijchen @doris-robot @xinyiZzz @liaoxin01 @yiguolei