Add configurable parameters for statsd client#16283
Merged
abhishekagarwal87 merged 3 commits intoapache:masterfrom Apr 17, 2024
Merged
Add configurable parameters for statsd client#16283abhishekagarwal87 merged 3 commits intoapache:masterfrom
abhishekagarwal87 merged 3 commits intoapache:masterfrom
Conversation
10 tasks
karthikgurram87
suggested changes
Apr 15, 2024
karthikgurram87
left a comment
karthikgurram87
left a comment
There was a problem hiding this comment.
There are queues and buffers in StatsDProcessor.java and StatsDSender.java. You have to increase their sizes as well.
hardikbajaj
changed the title
Comment on lines
+47
to
+50
| |`druid.emitter.statsd.queueSize`|Maximum number of unprocessed messages in the message queue.|no|Default value of StatsD Client(4096)| | ||
| |`druid.emitter.statsd.poolSize`|Network packet buffer pool size.|no|Default value of StatsD Client(512)| | ||
| |`druid.emitter.statsd.processorWorkers`|The number of processor worker threads assembling buffers for submission.|no|Default value of StatsD Client(1)| | ||
| |`druid.emitter.statsd.senderWorkers`| The number of sender worker threads submitting buffers to the socket.|no|Default value of StatsD Client(1)| |
Contributor
There was a problem hiding this comment.
are these the same defaults that statsd client gets created without your patch?
Contributor
Author
There was a problem hiding this comment.
Yes @abhishekagarwal87 . I've tested on a cluster with these values.
I've initialised them to always use the DEFAULTS of statsd client for these parameters
https://github.com/apache/druid/pull/16283/files#diff-3792ee2083cfb5ebd2255b8f65c9c8d7bf92a82325e48bb9ae59d252a5d292a2R99
abhishekagarwal87
approved these changes
Apr 17, 2024
hardikbajaj
added a commit
to confluentinc/druid
that referenced
this pull request
Apr 17, 2024
Statsd client sometimes drops metrics when this queueSize of statsd client with max unprocessed messages is completely full. This causes some high cardinality metrics like per partition lag being droppped. There are multiple parameters of statsdclient that can be initialized and can help increase the load/capacity of client to not to drop metrics more frequently. Properties like queueSize, poolSize, processorWorkers and senderWorkers will now be configurable at runtime
hardikbajaj
added a commit
to confluentinc/druid
that referenced
this pull request
Apr 18, 2024
Statsd client sometimes drops metrics when this queueSize of statsd client with max unprocessed messages is completely full. This causes some high cardinality metrics like per partition lag being droppped. There are multiple parameters of statsdclient that can be initialized and can help increase the load/capacity of client to not to drop metrics more frequently. Properties like queueSize, poolSize, processorWorkers and senderWorkers will now be configurable at runtime
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #XXXX.
Statsd client sometimes drops metrics when this queueSize of statsd client with max unprocessed messages is completely full. This causes some high cardinality metrics like per partition lag being droppped.
There are multiple parameters of statsdclient that can be initialized and can help increase the load/capacity of client to not to drop metrics more frequently.
Properties like
queueSize,poolSize,processorWorkersandsenderWorkerswill now be configurable at runtimeDescription
Fixed the bug ...
Renamed the class ...
Added a forbidden-apis entry ...
Release note
Key changed/added classes in this PR
MyFooOurBarTheirBazThis PR has: