Reduce hostd mem usage via event type filter#7009
Merged
hickeng merged 3 commits intovmware:releases/1.3.0from Dec 29, 2017
Merged
Reduce hostd mem usage via event type filter#7009hickeng merged 3 commits intovmware:releases/1.3.0from
hickeng merged 3 commits intovmware:releases/1.3.0from
Conversation
An alternative to increasing the collector page size. It will reduce the throughput to the event collector and hence reduce event misses. See issues vmware#6937 and vmware#6998
Hypothesis is that it was primarily the reconfigure events that were overwhelming the heap due to associated config spec. This has not been confirmed by code inspection or similar, but this commit aims to experientially determine the impact on hostd memory. Experiments have shown that a filtered collector with pagesize of 200 has the same memory profile across full CI as the 1.2.1 release code with an unfiltered pagesize of 25 (maxes out at approx 125000). Without the filtering, significant deviation was observed.
hickeng
requested review from
anchal-agrawal,
cgtexmex,
dougm and
mhagen-vmware
hickeng
changed the title
AngieCris
approved these changes
Dec 29, 2017
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick PR. Original: #7001
This bumps eventPageSize back to 200 to determine if the event type filter has significant effect on memory footprint of task history.
Having run full CI against this branch we saw a maximum of
124880which is directly inline with the numbers found for 1.2.1 full CI and an unfiltered pagesize of 25.The fact these two numbers are almost identical implies to me that we're not saturating the 25 pagesize in CI and in fact the number is dominated at this time by the taskRetention interval in hostd.
https://ci.vcna.io/vmware/vic/15511 is the CI build used for reference. There is a failure in the log test which looks like it's related to either a local stall or clock skew - will confirm once logs are uploaded.
@dougm has tests for the filtering in vmware/govmomi and I've confirmed that we're getting only the expected events delivered. The
docker psand out-of-band tests already exercise the function that events support.This is based off #7000 and should be merged instead of that, or after having dropped the govmomi portion of this PR.
Follow up for : #6886 and #6384