Inconsistent handling of Stat Prefix between StatsD sinks and prometheus when a Custom Stat Namespace is registered.

[karanvasnani]

Title: Inconsistent handling of stat prefix between StatsD stats sinks and prometheus when a Custom Stat Namespace is registered.

Description:
In #17357, support for specifying a Custom Stat Namespace was added for extensions and stat sinks to register a namespace to distinguish between user-defined stats and Envoy native stats. Additionally, a default namespace of "wasmcustom" was registered for Wasm extensions. With these changes the exposed StatsD metrics changed from envoy.<metric_name> to be envoy.wasmcustom.<metric_name> where "envoy." is the default prefix added for all stats in Envoy.

Also in #17357, the prometheus stats handler was updated to handle the prefix when a custom stat namespace is registered. Changes made were as follows:

• Check for a registered custom namespace and strip the namespace prefix if it is defined from the exposed metric.
• Also remove the default "envoy_" prefix for user-defined metrics.

Therefore, with that change the user-defined metrics are exposed as-is with no prefix from prometheus and the format therefore changed from envoy_<metrics_name> to just <metric_name>.

This has made the behavior between StatsD sinks and prometheus exposed metrics inconsistent. Ideally, similar changes should be made for StatsD to make the behavior consistent with prometheus.

Repro steps:

1. Requires a wasm stats extension with user-defined metrics
2. Query stats endpoint at 9901/stats and verify the stat format as wasmcustom.<user-defined-metric-name> with the "wasmcustom." prefix.
3. Use a statsD sink for exposing metrics e.g., DogStatsD.
4. Observe the exposed statsD stat is in the format envoy.wasmcustom.<user-defined-metric-name> with the additional "envoy." prefix.
5. Now query the prometheus endpoint at 9901/stats/prometheus to verify the stat format to be <user-defined-metric-name> without any prefix.

[RyanTheOptimist]

cc @jmarantz

[jmarantz]

I didn't remember the wasm stats had a prefix of "envoy.wasmcustom"; I thought it was just "wasmcustom" but I might not be remembering correctly as I just reviewed the PR and didn't write it.

Why is it necessary for statsd and prometheus to present the same set of names? I'm curious what problems occur due to an inconsistency between them.

[mathetake]

Why is it necessary for statsd and prometheus to present the same set of names? I'm curious what problems occur due to an inconsistency between them.

+1, and if it is necessary I could do a PR on statsd, but I would like to know the motivation better beforehand (as in the next few months I won't be able to have enough cycles for Envoy PRs).

[karanvasnani]

Sorry for the delay in responding. I agree that it shouldn't be necessary for statsd and prometheus to be consistent in stats naming. We actually have a Wasm stats extension that is included with our Envoy image and we rely on these stats to be consistent for providing a better experience for the users.
I think we can look at it as a feature request for StatsD rather than a bug as it makes sense in my opinion to introduce the similar handling for stat namespace prefix in statsd as was done in prometheus. By also reading the description on the #17357 it feels like the intention was to add "wasmcustom." prefix internally for distinguishing and allow stat sinks to determine how to handle exposing these metrics. Please let me know if that makes sense.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.