node_exporter uses excessive CPU on a ThunderX2 host

[djeverett]

Host operating system: output of uname -a

Linux rhel-8-2-aarch64-gpu-1 4.18.0-193.14.2.el8_2.aarch64 #1 SMP Wed Jul 29 19:46:42 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

node_exporter version: output of node_exporter --version

1.0.1

node_exporter command line flags

--collector.systemd --collector.textfile --collector.textfile.directory=/var/lib/node_exporter --collector.filesystem --collector.filesystem.ignored-fs-types='^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fuse..*|fusectl|hugetlbfs|iso9660|mqueue|nfs|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs|tmpfs)$' --collector.filesystem.ignored-mount-points='^/(dev|proc|run|sys)($|/)' --web.listen-address=0.0.0.0:9100

Are you running node_exporter in Docker?

No

What did you do that produced an error?

On a ThunderX2 machine, run node_exporter and top. The run curl localhost:9100/metrics and watch for node_exporter in top

The output of lscpu on this host is:

Architecture:        aarch64
Byte Order:          Little Endian
CPU(s):              256
On-line CPU(s) list: 0-255
Thread(s) per core:  4
Core(s) per socket:  32
Socket(s):           2
NUMA node(s):        2
Vendor ID:           Cavium
Model:               2
Model name:          ThunderX2 99xx
Stepping:            0x1
CPU max MHz:         2500.0000
CPU min MHz:         1000.0000
BogoMIPS:            400.00
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            32768K
NUMA node0 CPU(s):   0-127
NUMA node1 CPU(s):   128-255
Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics cpuid asimdrdm

What did you expect to see?

%CPU for node_exporter is a few percent.

What did you see instead?

CPU usage goes up to more than 1200%.

[uniemimu]

The cpu (-freq) collector(s) parallelize a lot, which can cause a sudden burst of threads accessing related kernel functionality pretty much at the same time. Could you try running node_exporter without the cpu* collectors to see if the issue is in those?

[djeverett]

Thanks for the response. I can confirm that the excessive CPU load is not seen if node_exporter is started with --no-collector.cpufreq. We are not actually using the cpufreq metrics, so disabling the collector is an appropriate solution for us.

[uniemimu]

A long time ago I recall suggesting not to do frequency querying for hyperthreaded cores. The frequency for those is always the same as for the physical core, so one could as well just copy the information from the related physical core value. That would on some CPUs reduce the thread burden by 50% and as such it would be a valid optimization. But I fear that is probably not quite a sufficient remedy.

Perhaps limiting the thread count to something much lower than the physical core count is the way to go. Or the related kernel side functionality should be investigated to find out what exactly is so problematic when many threads try to read the frequency information at the same time.

Last but not least I would like to emphasize that reading the frequency information with as many threads as possible is problematic also in the sense that on some hardware architectures it has a rather significant impact on the result, basically exhibiting a CS case of observer effect. Starting as many threads as there are cores just to read the frequency information "fast" will on certain high-core architectures immediately kick the cores out from any kind of turbo-frequencies they might have used, basically rendering the information that is read, rather useless. You'll never see those turbo frequencies reported with node_exporter, even if they in fact are getting used. I could also imagine overall CPU-frequencies going up (but not as far as to turbo) during the period of reading the information, if spinlocks are getting congested (which could explain the perceived excessive CPU usage). But of the latter, I do not have proof. For the lack of turbo-frequencies getting reported, that I know and have witnessed for sure on high core count machines.

@SuperQ please take note

[SuperQ]

The problem with limiting threads is it will cause the scrape to be slow and just defer the problem.

Remember, these are not OS threads, these are goroutines. If you want to limit the core time of the node_exporter, it's easier to use GOMAXPROCS or cgroup controls.

[uniemimu]

I can verify that this can be an issue also for certain high core count Xeons running short-ish scrape intervals, and the workaround is disabling of cpufreq collector.

I haven't seen any issues with low core count nodes.

[discordianfish]

Can y'all confirm this happens with both the cpu and the cpufreq collector?

Given that more people report this now, we need to do something about this. I suspect there is some kernel issue that we're triggering. We're saying:

	// Execute the parsing of each CPU in parallel.
	// This is done because the kernel intentionally delays access to each CPU by
	// 50 milliseconds to avoid DDoSing possibly expensive functions.

• https://github.com/prometheus/procfs/blob/master/sysfs/system_cpu.go#L164
  Have we confirmed that this is the case? Maybe the kernel delays it precisely because rapid querying this endpoint results in the problems we're saying.

[SuperQ]

IIRC, @rtreffer found the explicit 50ms delay in the kernel code.

[uniemimu]

Can y'all confirm this happens with both the cpu and the cpufreq collector?

For us disabling cpufreq-collector was sufficient to bring the load down and the nagging from the fellow maintaining one particular cluster ceased with that. We do still run the cpu-collector.

[SuperQ]

Thanks, I'm guessing this would be solved with GOMAXPROCS=1 as well.

[discordianfish]

@uniemimu Can you try setting GOMAXPROCS=1, enable the cpufreq collector and see if you still see the issue?

[SuperQ]

For reference, the original issue about cpufreq taking too long is here: #1086

[uniemimu]

@uniemimu Can you try setting GOMAXPROCS=1, enable the cpufreq collector and see if you still see the issue?

It is better, and doesn't get out of control anymore in a 96 core machine. However on an idling machine you can still easily tell that cpufreq is running, the 1m load went from 0.5 to 1.5 or like that. Compare with the default GOMAXPROCS the system becomes rather unstable and logging in with ssh is very sluggish, with the user quickly going for "sudo kill -9 pidof node_exporter". Load averages are in the hundreds then.