Envoy stats sink - flush time or max_bytes_per_datagram

[javedsha]

Title: How Envoy flushes stats to stats sink?

Description:
I have a dogstatsdsink set up and the CPU utlization of envoy and the sink is going very high, due to which throughput to redis cluster is affected. Without the stats sink, everything works perfectly (as I expect the throughput and latency to be of redis calls)

Is there a way to set the stats flush time in envoy ?
Or what is the ideal value for max_bytes_per_datagram?

config:
"stats_config": {
"use_all_default_tags": "true",
"stats_tags": [
{
"tag_name": "custom_namespace",
"fixed_value": "ch"
}
]
},
"stats_sinks": [
{
"name": "envoy.stat_sinks.dog_statsd",
"typed_config": {
"@type": "type.googleapis.com/envoy.config.metrics.v3.DogStatsdSink",
"prefix": "envoy",
"address": {
"socket_address": {
"protocol": "UDP",
"address": "127.0.0.1",
"port_value": xxxx
}
}
}
}
]

We are running envoy with conncurency 4 (have tried all concurrency setting, but same issue always).

The stats agent is taking 120% CPU, and is a bottleneck for redis calls. Without the stats configuration above, our redis performance increases close to double.

[optional Relevant Links:]

Any extra documentation required to understand the issue.

[alyssawilk]

cc @jmarantz

[javedsha]

@jmarantz Hello, ping :) This is a blocker for us at the moment, any help will be much appreciated. Thanks.

[jmarantz]

Have you captured a flame graph? I don't know DogStatsSink; is this for Data Dog?

[jmarantz]

To answer your specific question, you can set the stats flush interval using https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/bootstrap/v3/bootstrap.proto#envoy-v3-api-field-config-bootstrap-v3-bootstrap-stats-flush-interval

[vinay-D-law]

hi @jmarantz we tried that and increased the time to 20 secs.
"stats_flush_interval": "20s" but nothing changed , is this the right way to do, or do we need to write time in ms or in some other format.

[sschepens]

Something we realized recently is that Histograms in Statsd are submitted each time a value is recorded, because Statsd is the one aggregating and calculating the histogram.
This basically causes envoy to send several UDP packets to Statsd for each request.

[sschepens]

here is the code where statsd ends up flushing histograms everytime a value is recorded.

So flush interval really only apples to counter and gauges.

[jmarantz]

Yes that seems to be how it works. I'm a little surprised by this, to be honest, but it doesn't appear to be an accident.

Is the problem that there's no buffering in the UDP statsd filter, and we really should batch these up?

[sschepens]

We could batch these and attempt to send multiple in the same UDP packet.

Also, Dogstatsd (DataDogs implementation of Statsd) allows batching submissions of histogram values, it basically supports sending a list of values to submit for a histogram.

[javedsha]

is the batching possible today? or disabling the histogram?

[jmarantz]

There is a new DataDog interface. @dgoffredo FYI. See #23958

It might be that the statsd protocol is not the optimal one for populating Data Dog.

FYI we use this histogram hook internally to copy Envoy histogram data to our own custom histogram structures, effectively batching the updates.

[dgoffredo]

The work in #23958 was for the tracing subsystem, and does not involve the submission of statistics.