feat(provider): add External Metrics provider

[jlore-decathlon]

Proposed addition

The current Datadog metric provider relies on their Metric API.
However, this API has pretty low rate limits, and people with a moderately sized infrastructure tend to reach these limits quite easily when scaling their usage of Flagger or datadog-based autoscaling (like KEDA).

Datadog offers a more scalable alternative by making its Cluster Agent batch requests by groups of 35 see Cluster Agent Autoscaling Metrics. It then makes these metrics available within the cluster by exposing an endpoint following Kubernetes External Metrics API.

Note

This endpoint is not documented by Datadog, as they expect people to have the agent register against the control plane as the cluster's external metrics provider and then making these metrics available through k8s API Server, removing the need to query the endpoint directly.
However, by implementing a kubernetes API, its behavior is predictable and stable enough to be used directly.

We've relied on the way KEDA implemented a similar feature during design and implementation. However, Flagger is not an autoscaling solution so we're not going to mimic the metric proxy Keda operates. We simply propose to query the external metric server directly. By doing this, we also chose to make the provider generic and compatible with any external metrics server. The downside is that we cannot abstract the way datadog names its metrics which isn't trivial.

fix: #1235

Any alternatives you've considered?

We've pondered modifying the Datadog metric provider instead of making an external metrics provider. But we felt that this had the benefit of making other external metric providers compatible and kept the code datadog-agnostic.

We could theoretically make it even more generic and use any kubernetes metric API (standard, Custom or External), but I think Flagger already offers this

Disclaimer

• This PR was peer programmed with my colleague @mveroone.
• We're not Go developers, yet we did our best to follow the project's coding conventions and guidelines. Any feedback is welcome and we'll be happy to rework any part of that contribution you think needs it.
• AI disclosure : AIL 1 (see https://danielmiessler.com/blog/ai-influence-level-ail). Minor code autocomplete, but mostly manual coding and writing.
• We've built the docker image and tested end-to-end on one of our GKE clusters against Datadog cluster agent endpoint. it seems to work as expected, but the lack of feedback from Flagger leaves room for some doubts, but we were unsure if adding some logging was a good idea.

[aryan9600]

thank you for opening this PR!

[mveroone]

Note : if that's okay, we'll squash commits after a few rounds of review so we can fix the DCO

[aryan9600]

looking good - a few nits

[aryan9600]

lgtm! 🎖️

please squash this into 1-2 commits and sign them off, thanks

[mveroone]

lgtm! 🎖️

please squash this into 1-2 commits and sign them off, thanks

@aryan9600 Done !
Thanks for the support.

[aryan9600]

ci is failing because of unformatted code - could you run make fmt and push again? thanks

[jlore-decathlon]

ci is failing because of unformatted code - could you run make fmt and push again? thanks

@aryan9600 Done