rpc metrics

[gabreal]

There are currently no metrics for observing the load of RPC queries in the substrate/polkadot client. These would be very valuable to understand the load of an RPC node and to detect patterns and application behaviours.

Thus metrics could be added that would count the methods called (with an unknown method label for malformed requests). Additionally metrics for successful and erroneous requests would be good (with response codes). Timings of these RPC calls would be a plus but not be collected per default.

For keeping the number of metrics limited overall the exposure of such metrics could be made dependent on logging flags e.g. -l rpc=verbose.

[dvdplm]

/cc @niklasad1 @maciejhirsz

[niklasad1]

Yeah, I totally agree with these metrics and I think we need these metrics make for paritytech/jsonrpsee#203 to work properly long-term.

[jak-pan]

As mentioned in #8774 it would be cool to have also number of open sockets to get a grasp of the type of users connecting to RPCs

[gilescope]

Improbable had a good selection of generic service metrics over their endpoints that all pumped into prometheus: https://github.com/grpc-ecosystem/go-grpc-prometheus

[lovelaced]

Bump on this - it's becoming more important we're able to monitor this due to increased resource usage, and would also be interesting to see usage patterns. Just count would be plenty for now, and would let us make more informed decisions about scaling.