The value of CPU usage is negative

[brandonwusc]

Host operating system: output of uname -a

Linux localhost 4.19.90-rt35 #1 SMP PREEMPT RT Wed Jun 19 19:51:10 CST 2024 aarch64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.8.2 (branch: HEAD, revision: f1e0e83)
build user: root@03d440803209
build date: 20240714-11:54:16
go version: go1.22.5
platform: linux/arm64
tags: unknown

node_exporter command line flags

/usr/local/node_exporter/node_exporter --collector.textfile.directory=/tmp/node_exporter/

node_exporter log output

-sh-4.4# ts=2024-09-02T05:14:33.021Z caller=node_exporter.go:193 level=info msg="Starting node_exporter" version="(version=1.8.2, branch=HEAD, revision=f1e0e8360aa60b6cb5e5cc1560bed348fc2c1895)"
ts=2024-09-02T05:14:33.021Z caller=node_exporter.go:194 level=info msg="Build context" build_context="(go=go1.22.5, platform=linux/arm64, user=root@03d440803209, date=20240714-11:54:16, tags=unknown)"
ts=2024-09-02T05:14:33.022Z caller=node_exporter.go:196 level=warn msg="Node Exporter is running as root user. This exporter is designed to run as unprivileged user, root is not required."
ts=2024-09-02T05:14:33.024Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/)
ts=2024-09-02T05:14:33.024Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
ts=2024-09-02T05:14:33.026Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(z?ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$
ts=2024-09-02T05:14:33.027Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data
ts=2024-09-02T05:14:33.028Z caller=node_exporter.go:111 level=info msg="Enabled collectors"
ts=2024-09-02T05:14:33.029Z caller=node_exporter.go:118 level=info collector=arp
ts=2024-09-02T05:14:33.029Z caller=node_exporter.go:118 level=info collector=bcache
ts=2024-09-02T05:14:33.029Z caller=node_exporter.go:118 level=info collector=bonding
ts=2024-09-02T05:14:33.030Z caller=node_exporter.go:118 level=info collector=btrfs
ts=2024-09-02T05:14:33.030Z caller=node_exporter.go:118 level=info collector=conntrack
ts=2024-09-02T05:14:33.030Z caller=node_exporter.go:118 level=info collector=cpu
ts=2024-09-02T05:14:33.030Z caller=node_exporter.go:118 level=info collector=cpufreq
ts=2024-09-02T05:14:33.031Z caller=node_exporter.go:118 level=info collector=diskstats
ts=2024-09-02T05:14:33.031Z caller=node_exporter.go:118 level=info collector=dmi
ts=2024-09-02T05:14:33.031Z caller=node_exporter.go:118 level=info collector=edac
ts=2024-09-02T05:14:33.031Z caller=node_exporter.go:118 level=info collector=entropy
ts=2024-09-02T05:14:33.032Z caller=node_exporter.go:118 level=info collector=fibrechannel
ts=2024-09-02T05:14:33.032Z caller=node_exporter.go:118 level=info collector=filefd
ts=2024-09-02T05:14:33.032Z caller=node_exporter.go:118 level=info collector=filesystem
ts=2024-09-02T05:14:33.032Z caller=node_exporter.go:118 level=info collector=hwmon
ts=2024-09-02T05:14:33.033Z caller=node_exporter.go:118 level=info collector=infiniband
ts=2024-09-02T05:14:33.033Z caller=node_exporter.go:118 level=info collector=ipvs
ts=2024-09-02T05:14:33.033Z caller=node_exporter.go:118 level=info collector=loadavg
ts=2024-09-02T05:14:33.034Z caller=node_exporter.go:118 level=info collector=mdadm
ts=2024-09-02T05:14:33.034Z caller=node_exporter.go:118 level=info collector=meminfo
ts=2024-09-02T05:14:33.034Z caller=node_exporter.go:118 level=info collector=netclass
ts=2024-09-02T05:14:33.034Z caller=node_exporter.go:118 level=info collector=netdev
ts=2024-09-02T05:14:33.035Z caller=node_exporter.go:118 level=info collector=netstat
ts=2024-09-02T05:14:33.035Z caller=node_exporter.go:118 level=info collector=nfs
ts=2024-09-02T05:14:33.035Z caller=node_exporter.go:118 level=info collector=nfsd
ts=2024-09-02T05:14:33.035Z caller=node_exporter.go:118 level=info collector=nvme
ts=2024-09-02T05:14:33.036Z caller=node_exporter.go:118 level=info collector=os
ts=2024-09-02T05:14:33.036Z caller=node_exporter.go:118 level=info collector=powersupplyclass
ts=2024-09-02T05:14:33.036Z caller=node_exporter.go:118 level=info collector=pressure
ts=2024-09-02T05:14:33.036Z caller=node_exporter.go:118 level=info collector=rapl
ts=2024-09-02T05:14:33.037Z caller=node_exporter.go:118 level=info collector=schedstat
ts=2024-09-02T05:14:33.037Z caller=node_exporter.go:118 level=info collector=selinux
ts=2024-09-02T05:14:33.037Z caller=node_exporter.go:118 level=info collector=sockstat
ts=2024-09-02T05:14:33.037Z caller=node_exporter.go:118 level=info collector=softnet
ts=2024-09-02T05:14:33.038Z caller=node_exporter.go:118 level=info collector=stat
ts=2024-09-02T05:14:33.039Z caller=node_exporter.go:118 level=info collector=tapestats
ts=2024-09-02T05:14:33.043Z caller=node_exporter.go:118 level=info collector=textfile
ts=2024-09-02T05:14:33.044Z caller=node_exporter.go:118 level=info collector=thermal_zone
ts=2024-09-02T05:14:33.044Z caller=node_exporter.go:118 level=info collector=time
ts=2024-09-02T05:14:33.053Z caller=node_exporter.go:118 level=info collector=timex
ts=2024-09-02T05:14:33.056Z caller=node_exporter.go:118 level=info collector=udp_queues
ts=2024-09-02T05:14:33.058Z caller=node_exporter.go:118 level=info collector=uname
ts=2024-09-02T05:14:33.060Z caller=node_exporter.go:118 level=info collector=vmstat
ts=2024-09-02T05:14:33.063Z caller=node_exporter.go:118 level=info collector=watchdog
ts=2024-09-02T05:14:33.065Z caller=node_exporter.go:118 level=info collector=xfs
ts=2024-09-02T05:14:33.066Z caller=node_exporter.go:118 level=info collector=zfs
ts=2024-09-02T05:14:33.070Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9100
ts=2024-09-02T05:14:33.070Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9100

Are you running node_exporter in Docker?

No.

What did you do that produced an error?

Nothing,

What did you expect to see?

The usage of CPU should never be negative

What did you see instead?

The CPU3 sometimes shows negative values.

Grafana(10.41.3.187) cmd, screenshot and pcap:
100 - sum by(instance) (irate(node_cpu_seconds_total{instance="10.41.5.101:9100", job="file_sd_targets", mode="idle", cpu="3"}[$__rate_interval])) * 100
100 - sum by(instance) (irate(node_cpu_seconds_total{instance="10.41.5.101:9100", job="file_sd_targets", mode="idle", cpu="3"}[$__rate_interval])) * 100

[brandonwusc]

182_grafana.zip

[SuperQ]

For questions/help/support please use our community channels. There are more people available to potentially respond to your request and the whole community can benefit from the answers provided.