Skip to content

Utility for collecting new gauge metrics#9017

Merged
mgritter merged 10 commits into
masterfrom
new_metrics_2
May 19, 2020
Merged

Utility for collecting new gauge metrics#9017
mgritter merged 10 commits into
masterfrom
new_metrics_2

Conversation

@mgritter
Copy link
Copy Markdown
Contributor

This module takes care of:

  • Running a collection process periodically
  • Backing off the collection interval if it takes too long to collect all gauges
  • Limiting the cardinality (of each iteration) and streaming the collected information to go-metrics

@mgritter mgritter requested review from mjarmy, ncabatoff and sgmiller May 18, 2020 01:36
@mgritter mgritter added this to the 1.5 milestone May 18, 2020
ncabatoff
ncabatoff previously approved these changes May 18, 2020
Copy link
Copy Markdown
Collaborator

@ncabatoff ncabatoff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice PR with great comments and tests. I have a bunch of minor suggestions, none of which are blockers.

// so that collection processes do not all run at the time time.
// If we knew all the procsses in advance, we could just schedule them
// evenly, but a new one could be added per secret engine.
func (p *GaugeCollectionProcess) delayStart() bool {
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe rename to delayOrStop to hint at how to interpret its return value?

Comment thread helper/metricsutil/gauge_process.go Outdated
Comment thread helper/metricsutil/gauge_process.go Outdated
Comment thread helper/metricsutil/gauge_process.go
}

if err != nil {
p.logger.Error("error collecting gauge", "id", p.labels, "error", err)
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I worry a little about this log message being too spammy. Do we want to use lower severity? Throttle? Replace with an error metric?

Personally regardless of whether we change the logging, I like having error metrics for my metrics collections. That way when creating a dashboard or alert rule, I can easily take into account if the metrics are dubious due to a collection error.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added an error count.

I think the frequency of error messages will not be too high to worry about, in this case, but it might become a concern if there is a storage outage. I suspect there will be many more error messages in that case anyway?

s.tickerBarrier = make(chan *SimulatedTicker, n)
}

func startSimulatedTime() *SimulatedTime {
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a constructor, why not call it newSimulatedTime?

Comment thread helper/metricsutil/gauge_process.go Outdated
Comment thread helper/metricsutil/gauge_process_test.go
Comment thread helper/metricsutil/gauge_process_test.go
Comment thread helper/metricsutil/gauge_process_test.go
Co-authored-by: ncabatoff <ncabatoff@hashicorp.com>
mgritter added 4 commits May 18, 2020 17:40
  * Name change for constructor
  * Allow cancel while streaming stats.
  * Make Done a function call rather than a public data member.
@mgritter mgritter merged commit 72afe55 into master May 19, 2020
@mgritter mgritter deleted the new_metrics_2 branch June 24, 2020 00:49
sapk pushed a commit to sapk-fork/vault that referenced this pull request Sep 1, 2025
…ashicorp#9017) (hashicorp#9019)

Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants