Dogstatsd and statsd `func (g *Gauge) Add()` produces incorrect values across flushes

[jfhamlin]

Both Add() implementations update observations using lv.node's add, but the space is reset on every flush:

• https://github.com/go-kit/kit/blob/master/metrics/statsd/statsd.go#L131
• https://github.com/go-kit/kit/blob/master/metrics/dogstatsd/dogstatsd.go#L139

Following the documentation quoted below, a negative aggregate delta over a flush period is interpreted by statsd as a delta when sent, whereas a positive aggregate delta, as sent, is interpreted as an absolute value.

For example, assume we call gauge.Add(1) once per flush period, with no other calls to Add or Set for that gauge. The resulting metric will have a constant value of 1; the additions will not accumulate.

These gauges either need to track and send the absolute value (noting the caveat for negative values quoted below) or include the sign character on all flushed values to denote deltas.

https://github.com/etsy/statsd/blob/master/docs/metric_types.md#gauges:

Adding a sign to the gauge value will change the value, rather than setting it.

gaugor:-10|g
gaugor:+4|g
So if gaugor was 333, those commands would set it to 333 - 10 + 4, or 327.

Note:

This implies you can't explicitly set a gauge to a negative number without first setting it to zero.

[peterbourgon]

The easier path is to take the sign option.

[jfhamlin]

From DataDog support:

So I have confirmed that we do not support gauge deltas.

So the dogstatsd implementation of Gauge will have to track the gauge value. I'm happy to work on a fix for this, but I don't have a firm ETA.

[pbitty]

Looks like this was fixed in #588.

[peterbourgon]

🙇