Skip to content

[fix](memory) Fix BE memory info compatible with Cgroup #35412

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
May 27, 2024
Merged

[fix](memory) Fix BE memory info compatible with Cgroup #35412

morningman merged 1 commit into from
May 27, 2024

Conversation

@xinyiZzz
Copy link
Contributor

@xinyiZzz xinyiZzz commented May 26, 2024 •
edited
Loading

Proposed changes

  1. memory.usage_in_bytes ~= free.used + free.(buff/cache) - (buff), free cache can be reused,
    so, modify cgroup_memory_usage = memory.usage_in_bytes - memory.meminfo["Cached"].
  2. If system not configured with cgroup, find cgroup file path will failed, refactor refresh cgroup memory info, compatible with find failed.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented May 26, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented May 26, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented May 26, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented May 26, 2024

TPC-H: Total hot run time: 41403 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c567fa8b79303374fe272a7cdf7f7d1a662485b1, data reload: false

------ Round 1 ----------------------------------
q1	17597	4402	4312	4312
q2	2012	197	194	194
q3	10457	1241	1243	1241
q4	10216	790	863	790
q5	7501	2747	2726	2726
q6	230	135	135	135
q7	968	615	600	600
q8	9235	2148	2115	2115
q9	9611	6778	6826	6778
q10	9413	3842	3975	3842
q11	485	247	253	247
q12	468	227	225	225
q13	17442	3188	3204	3188
q14	262	223	213	213
q15	509	461	475	461
q16	500	431	399	399
q17	989	702	717	702
q18	8423	7931	7777	7777
q19	4458	1579	1522	1522
q20	663	326	313	313
q21	5159	3339	3949	3339
q22	348	284	285	284
Total cold run time: 116946 ms
Total hot run time: 41403 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4544	4418	4432	4418
q2	378	263	257	257
q3	3146	2906	2922	2906
q4	1875	1630	1619	1619
q5	5438	5511	5503	5503
q6	211	124	125	124
q7	2190	1824	1832	1824
q8	3255	3382	3415	3382
q9	8650	8701	8733	8701
q10	3989	3687	3816	3687
q11	585	507	494	494
q12	812	628	625	625
q13	16150	3143	3187	3143
q14	299	284	293	284
q15	538	472	483	472
q16	483	429	424	424
q17	1790	1495	1449	1449
q18	7682	7816	7505	7505
q19	1696	1520	1581	1520
q20	2027	1810	1767	1767
q21	4894	4757	4771	4757
q22	598	481	501	481
Total cold run time: 71230 ms
Total hot run time: 55342 ms

@doris-robot
Copy link

doris-robot commented May 26, 2024

TPC-DS: Total hot run time: 169391 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c567fa8b79303374fe272a7cdf7f7d1a662485b1, data reload: false

query1	911	386	373	373
query2	6438	2350	2360	2350
query3	6648	209	209	209
query4	19209	17382	17318	17318
query5	4143	414	417	414
query6	240	161	153	153
query7	4594	295	301	295
query8	252	197	181	181
query9	8468	2409	2392	2392
query10	457	274	290	274
query11	10682	10200	10061	10061
query12	144	88	85	85
query13	1629	364	358	358
query14	9434	6799	6897	6799
query15	206	168	169	168
query16	7412	255	253	253
query17	1330	516	524	516
query18	1851	282	268	268
query19	204	153	148	148
query20	90	86	88	86
query21	196	135	136	135
query22	4312	4206	4104	4104
query23	33776	33021	32978	32978
query24	7736	2834	2851	2834
query25	589	352	365	352
query26	702	154	151	151
query27	2192	317	320	317
query28	5297	2083	2097	2083
query29	844	595	606	595
query30	260	172	176	172
query31	971	772	746	746
query32	88	55	54	54
query33	534	280	281	280
query34	872	477	494	477
query35	713	642	605	605
query36	1047	941	898	898
query37	108	74	70	70
query38	2907	2788	2763	2763
query39	850	797	822	797
query40	192	127	121	121
query41	45	42	45	42
query42	102	97	99	97
query43	578	557	541	541
query44	1073	734	760	734
query45	176	164	162	162
query46	1069	740	723	723
query47	1861	1790	1831	1790
query48	364	289	298	289
query49	844	371	392	371
query50	778	381	385	381
query51	6930	6786	6677	6677
query52	109	87	97	87
query53	345	283	282	282
query54	559	422	418	418
query55	77	74	72	72
query56	286	254	233	233
query57	1125	1057	1074	1057
query58	225	210	209	209
query59	3459	2948	2967	2948
query60	271	257	252	252
query61	90	85	85	85
query62	590	468	441	441
query63	313	280	283	280
query64	8461	2190	1739	1739
query65	3145	3056	3105	3056
query66	806	323	322	322
query67	15260	15233	14898	14898
query68	4492	533	536	533
query69	435	261	259	259
query70	1132	1030	1139	1030
query71	370	264	267	264
query72	7461	5648	2745	2745
query73	728	326	319	319
query74	6028	5630	5518	5518
query75	3318	2640	2651	2640
query76	2345	1020	1012	1012
query77	397	263	267	263
query78	10237	9850	10100	9850
query79	1989	522	517	517
query80	1116	436	426	426
query81	539	242	244	242
query82	614	95	91	91
query83	245	176	170	170
query84	247	88	82	82
query85	1692	277	268	268
query86	510	286	322	286
query87	3308	3132	3122	3122
query88	4170	2348	2341	2341
query89	469	377	390	377
query90	2041	185	184	184
query91	121	99	94	94
query92	62	47	48	47
query93	2539	527	505	505
query94	1144	190	178	178
query95	404	306	303	303
query96	585	272	261	261
query97	3197	3005	3078	3005
query98	251	226	209	209
query99	1140	858	831	831
Total cold run time: 260795 ms
Total hot run time: 169391 ms

@doris-robot
Copy link

doris-robot commented May 26, 2024

ClickBench: Total hot run time: 31.48 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c567fa8b79303374fe272a7cdf7f7d1a662485b1, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.05
query3	0.23	0.05	0.05
query4	1.69	0.09	0.08
query5	0.51	0.49	0.50
query6	1.11	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.53	0.48	0.47
query10	0.54	0.54	0.53
query11	0.16	0.11	0.11
query12	0.14	0.12	0.11
query13	0.60	0.58	0.59
query14	0.78	0.78	0.78
query15	0.83	0.84	0.80
query16	0.36	0.36	0.37
query17	0.99	0.95	0.93
query18	0.20	0.22	0.23
query19	1.75	1.76	1.74
query20	0.01	0.01	0.02
query21	15.52	0.69	0.66
query22	4.37	6.41	2.91
query23	18.31	1.32	1.32
query24	1.33	0.33	0.28
query25	0.14	0.08	0.08
query26	0.27	0.16	0.17
query27	0.07	0.08	0.07
query28	13.38	1.00	1.00
query29	13.54	3.32	3.23
query30	0.24	0.06	0.05
query31	2.88	0.37	0.38
query32	3.30	0.47	0.46
query33	2.90	2.98	2.80
query34	17.17	4.46	4.48
query35	4.55	4.50	4.59
query36	0.67	0.45	0.47
query37	0.17	0.15	0.14
query38	0.15	0.14	0.15
query39	0.04	0.04	0.04
query40	0.16	0.14	0.15
query41	0.10	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.03	0.04
Total cold run time: 109.97 s
Total hot run time: 31.48 s

@doris-robot
Copy link

doris-robot commented May 26, 2024

TeamCity be ut coverage result:
Function Coverage: 35.65% (9015/25291)
Line Coverage: 27.33% (74571/272854)
Region Coverage: 26.55% (38579/145313)
Branch Coverage: 23.42% (19666/83984)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c567fa8b79303374fe272a7cdf7f7d1a662485b1_c567fa8b79303374fe272a7cdf7f7d1a662485b1/report/index.html

yiguolei
yiguolei approved these changes May 27, 2024
View reviewed changes
Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 27, 2024
@github-actions
Copy link
Contributor

github-actions bot commented May 27, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented May 27, 2024

PR approved by anyone and no changes requested.

morningman
morningman approved these changes May 27, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 31c3806 into apache:master May 27, 2024
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request May 27, 2024
@xinyiZzz
[fix](memory) Fix BE memory info compatible with Cgroup (apache#35412)
ba8f54c 
1. `memory.usage_in_bytes ~= free.used + free.(buff/cache) - (buff)`, free cache can be reused,
   so, modify cgroup_memory_usage = memory.usage_in_bytes - memory.meminfo["Cached"].
2. If system not configured with cgroup, find cgroup file path will failed, refactor refresh cgroup memory info, compatible with find failed.
@xinyiZzz xinyiZzz mentioned this pull request May 27, 2024
Merged
yiguolei pushed a commit that referenced this pull request May 27, 2024
@xinyiZzz
[fix](memory) Fix BE memory info compatible with Cgroup (#35412) (#35425
b6eaf95 
)

1. `memory.usage_in_bytes ~= free.used + free.(buff/cache) - (buff)`, free cache can be reused,
   so, modify cgroup_memory_usage = memory.usage_in_bytes - memory.meminfo["Cached"].
2. If system not configured with cgroup, find cgroup file path will failed, refactor refresh cgroup memory info, compatible with find failed.
seawinde pushed a commit to seawinde/doris that referenced this pull request May 27, 2024
@xinyiZzz @seawinde
[fix](memory) Fix BE memory info compatible with Cgroup (apache#35412)
f87e6f5 
1. `memory.usage_in_bytes ~= free.used + free.(buff/cache) - (buff)`, free cache can be reused,
   so, modify cgroup_memory_usage = memory.usage_in_bytes - memory.meminfo["Cached"].
2. If system not configured with cgroup, find cgroup file path will failed, refactor refresh cgroup memory info, compatible with find failed.
dataroaring pushed a commit that referenced this pull request May 28, 2024
@xinyiZzz @dataroaring
[fix](memory) Fix BE memory info compatible with Cgroup (#35412)
fb0e47c 
1. `memory.usage_in_bytes ~= free.used + free.(buff/cache) - (buff)`, free cache can be reused,
   so, modify cgroup_memory_usage = memory.usage_in_bytes - memory.meminfo["Cached"].
2. If system not configured with cgroup, find cgroup file path will failed, refactor refresh cgroup memory info, compatible with find failed.
@xinyiZzz xinyiZzz mentioned this pull request Jun 12, 2024
Closed
3 tasks
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Jul 2, 2024
@w41ter @gavinchou
(selectdb-cloud) Pick some bugfix from apache/doris apache#35412 apac…
32d70f4 
…he#35083 apache#35557 (apache#3405)

[fix](memory) Fix BE memory info compatible with Cgroup (apache#35412)
[fix](memory) Avoid frequently refresh cgroup memory info (apache#35083)
[fix](libjdk) Revert support loading libjvm at runtime (apache#35557)
johnpyp added a commit to johnpyp/doris-website that referenced this pull request Aug 7, 2024
@johnpyp
[doc] document enable_use_cgroup_memory_info
4b69fa6 
Introduced in this PR: apache/doris#35412
@johnpyp johnpyp mentioned this pull request Aug 7, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.4-merged dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@xinyiZzz @doris-robot @morningman @yiguolei @gavinchou