Conntrack kernel stats

[carlpett]

This PR adds conntrack kernel statistics to the conntrack collector, similar to the data obtained with conntrack -S. Of extra interest is the insert_failed counter, which for example allows monitoring for some situations such as kubernetes#56903.

This does require CAP_NET_ADMIN, however, so I've put these metrics behind a flag, --collector.conntrack.kernel-stats.

[SuperQ]

/cc @rtreffer @matthiasr I know you two have run into conntrack issues before.

I'm not sure we want features in the node_exporter that require CAP_NET_ADMIN. We have a policy to keep this exporter unprivileged. We need to consider this carefully, since it starts to make the exporter a privilege escalation target.

[rtreffer]

We had our own fair share of conntrack / DNS interaction, too. We ended up patching the UDP timeout in the kernel.

We are alerting on 100*node_nf_conntrack_entries / node_nf_conntrack_entries_limit > 90 (paging) and 100*node_nf_conntrack_entries / node_nf_conntrack_entries_limit > 50 (warning).

Are there cases where the conntrack table would be low but inserts still fail?

Given the kernel config CONFIG_NF_CONNTRACK_PROCFS there is also a /proc/net/stat/nf_conntrack containing the same data, world readable. However that option is usually disabled (and deprecated iirc - reading /proc/net/nf_conntrack which is controlled by the same setting is a bad idea due to a big lock).

Overall: monitoring the conntrack table is very useful as an overflow is a severe issue and hard to detect.

[discordianfish]

@rtreffer Curious about your kernel patch/fixes.. Maybe drop a note in kubernetes/kubernetes#56903 ?

I also ran into this DNS issue without a fix, none of the workarounds helped. So yeah I find these metrics important but also not sure about adding stuff that needs elevated capabilities. Given this isn't the first time we'd need them, maybe we should revisit the 'no capabilities' requirement to 'no root' and all defaults need not to require capabilities?

[discordianfish]

That being said, currently I'm using this textfile collector for the insert fails which works fine too:

#!/bin/bash
set -euo pipefail

conntrack -S | while read -ra parts; do
  IFS='=' read -ra cpu <<< "${parts[0]}"
  for m in "${parts[@]:1}"; do
    IFS='=' read -ra kv <<< "$m"
    echo "conntrack_${kv[0]}_total{cpu=\"${cpu[1]}\"} ${kv[1]}"
  done
done

[carlpett]

@rtreffer The "insert failed" state isn't actually related to the connection tracking table being full, but rather a race condition within the kernel. More info in the linked kubernetes info.

@discordianfish Your textfile collector does basically the same thing, yes.

[SuperQ]

Yea, while I like the idea of this collector, I don't think we want to have the exporter use CAP_NET_ADMIN. Perhaps we can find someone in the Linux kernel team that can help us find a way to get this data without root.

[discordianfish]

@SuperQ Yeah wondering about the reason for requiring capabilities for gathering this. Not sure what the kernel policy for these things are.

[carlpett]

It is possible to use netlink in general without CAP_NET_ADMIN (the wifi collector does so, for example), so probably what is needed is that the kernel has to start handling the different interfaces that the netfilter bits exposes differently, depending on if they are mutating/sensitive or not.
No idea how big/small this change is, though.

[SuperQ]

@rtreffer Also, wasn't there some kernel patches for conntrack table size sysctl settings that are not inherited? Or did that get fixed in Docker now?

[discordianfish]

So yeah, to include this we'd need the kernel to provide the info to unprivileged processes. Is that someone is up to bringing up and tracking it? If not, I think we should close the PR for now.

[ti-mo]

(library creator here) The conntrack nfnetlink family requires NET_ADMIN if you want to as much as dial into it. Since that family can be used to manipulate the conntrack table, conntrack stats would need to be split off into a different family for this to be reasonably secure.

As much as I'd be honored to have my lib being used by the exporter, I would opt to get the stats out of procfs instead, because it will just be available to everybody without requiring root. I predict this would be the response on LKML in case anyone would bother to ask there. :)

[carlpett]

Well then, seems like there isn't much to do then. At least I got to learn a bit about netlink in the process :)