Metrics crate integration with Grafana data source

[expenses]

The metrics crate provides a metrics-gathering facade similar to log. It would be great if we could integrate this with the Grafana data server from #3913, as we'd be able to easily switch it out for the prometheus observer or otherwise.

I'll probably end up doing this, but it should be fairly straightforward if anyone else wants to have a go.

[pepyakin]

I guess this might be related
сс @mattrutherford

[mattrutherford]

Yes - looks like there is probably a lot of overlap with the tracing library that we just started to integrate #4132

Was thinking about adding a Receiver for substrate-tracing that can send to your Grafana data server.

Might be worth having a brain-storm some time to align on strategy for this stuff?

[expenses]

Yup!

[expenses]

I think we probably want to integrate the telemetry system into this as well.

[marcio-diaz]

From validators room:

anyone working on prometheus for polkadot? the current json datasource doesnt allow us to configure alert in grafana

[tomaka]

Now that Prometheus is well in place, I suppose we can close this issue?
Feel free to reopen if not.