Track delta metrics between scrapes

[kgeckhart]

This PR introduces support for aggregating DELTA metrics in-memory. This implementation is somewhat based on #167 but with extra functionality which is intended to solve the problem identified here, #167 (comment).

I chose to inject interfaces for the delta and distribution stores at the Collector level to try to keep library friendliness intact, #157

I did as much as I could to explain the solution and the tradeoffs with aggregating deltas in this manner in the README.md. The bulk of the changes are in monitoring_metrics.go where I tried to obfuscate as much of the delta logic as possible.

Should hopefully resolve #116 and #25

[SuperQ]

Thanks, I'll take a look at this soon.

To start, this needs a DCO sign-off. You can use git commit -s --amend to add it.

[ahjmorton]

Hello @SuperQ / @kgeckhart, this looks like it solves a really important issue with the exporter where DELTA metrics seem pretty much unusable in certain cases.

Can I ask what is stopping it from being merged and if there is anything I can do to speed that up? This issue is making monitoring of some of our GCP dependencies a lot harder as the data is split into 2 places.

[kgeckhart]

Hi @ahjmorton, I'll defer to @SuperQ for the review + merge side of things. I can say we have been running this version in production for a little over three weeks. Our largest exporter is tracking ~500k metrics without any noticeable issues.

[SuperQ]

Minor nits, but otherwise LGTM.

[SuperQ]

CC @igorwwwwwwwwwwwwwwwwwwww, this may interest you.

[igorwwwwwwwwwwwwwwwwwwww]

cc @T4cC0re

[SuperQ]

LGTM