Collect ENA Metrics

[sidewinder12s]

Can node_exporter collect these new metrics from the latest ENA Driver version?

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-network-performance-ena.html

Not sure how they are exposed other than through ethtool

[SuperQ]

Interesting idea. I don't know about how to get it other than ethtool either. We don't allow external processes in the node_exporter, so we'd have to look at how ethtool gets the data. It looks like it's all direct ioctls to the kernel.

I found one ethtool Go library that seems to implement the right ioctls: https://github.com/safchain/ethtool

[steven-aerts]

Guess this issue is related to #1832

[sidewinder12s]

Ya, sounds like these metrics would get pulled in by #1832

On an instance you can see the Nitro specific metrics with ethtool -S <interface id>

[SuperQ]

I like the idea of adding this, but I think we need a bit more effort into #1832. Having a tool that dumps raw data is useful, but hard to use for users. I would prefer to have the collector understand the data being read, and produce more useful metric names.

[sidewinder12s]

Appears #1832 is merged and lays the groundwork for getting these metrics. However it maintains a specific list of metrics it'll expose out of ethtool instead of just dumping everything, so we'll need to add the ENA specific metrics.

They are:

# Actual performance metrics
     bw_in_allowance_exceeded: 927679
     bw_out_allowance_exceeded: 0
     pps_allowance_exceeded: 154586
     conntrack_allowance_exceeded: 0
     linklocal_allowance_exceeded: 0

# ENA driver metrics
     ena_admin_q_aborted_cmd: 0
     ena_admin_q_submitted_cmd: 53
     ena_admin_q_completed_cmd: 53
     ena_admin_q_out_of_space: 0
     ena_admin_q_no_completion: 0

I'm not sure if I'll have time/able to dig into it but wanted to update the issue context. Also not sure how the existing code would work with metrics that may not exist (since these would only be available on AWS EC2 with the ENA drivers installed)

[discordianfish]

Maybe we could add a regex to configure which ethtool stats to return?

[sidewinder12s]

Is there prior art for that kind of configuration for node_exporter? Seems fine to me otherwise as long as it's configurable enough since a lot of these metrics may not match a single pattern.

I know there is another issue (#2096) about these ethtool metrics being extremely loud that might benefit from some configurability.

[discordianfish]

@sidewinder12s We have several regex flags to filter for devices etc. I don't think we have any that would select metrics but it seems reasonable to me.
@SuperQ wdyt?

[discordianfish]

I've got confused, the ethtool collector already exposes all ethtool metrics. So this here is already implemented, the stats should be there.

[discordianfish]

(I've just submitted #2117 though to filter them down)

[sidewinder12s]

I'll reopen if that's not the case, I had tried the latest released image and it didn't appear to expose the ENA metrics.

[sidewinder12s]

Awesome, can confirm they are there with latest! Must of missed them in the pile of metrics that ethtool exposes. Filtering would be very good.

[discordianfish]

I've added filtering in #2117