ZFS arcstat `memory_available_bytes` is missing on Linux

[gertvdijk]

Host operating system: output of uname -a

Linux 5.15.0-60-generic #66-Ubuntu SMP Fri Jan 20 14:29:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

$ node_exporter --version
node_exporter, version 1.5.0 (branch: HEAD, revision: 1b48970ffcf5630534fb00bb0687d73c66d1c959)
  build user:       root@6e7732a7b81b
  build date:       20221129-18:59:09
  go version:       go1.19.3
  platform:         linux/amd64

node_exporter command line flags

Node exporter command line flags

ExecStart=/usr/local/bin/node_exporter \
--collector.diskstats \
    --collector.diskstats.ignored-devices='^(dm-|mapper/opened:|ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\\d+n\\d+p)\\d+$' \
--collector.textfile \
    --collector.textfile.directory=/var/tmp/node_exporter \
--collector.filesystem \
    --collector.filesystem.fs-types-exclude='^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tmpfs|tracefs)$' \
    --collector.filesystem.mount-points-exclude='^/(sys|proc|dev|srv/somedir)($|/)' \
--collector.slabinfo \
--collector.systemd \
    --collector.systemd.unit-exclude='.+\\.(device|mount|scope|socket)$' \
    --no-collector.bonding \
    --no-collector.bcache \
    --no-collector.fibrechannel \
    --no-collector.infiniband \
    --no-collector.ipvs \
    --no-collector.nfs \
    --no-collector.nvme \
    --no-collector.pressure \
    --no-collector.rapl \
    --no-collector.tapestats \
    --web.listen-address=0.0.0.0:9100 \
    --web.telemetry-path=/metrics

node_exporter log output

Node exporter logs

ts=2023-04-05T14:40:24.356Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)"
ts=2023-04-05T14:40:24.356Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)"
ts=2023-04-05T14:40:24.356Z caller=systemd_linux.go:152 level=info collector=systemd msg="Parsed flag --collector.systemd.unit-include" flag=.+
ts=2023-04-05T14:40:24.356Z caller=systemd_linux.go:154 level=info collector=systemd msg="Parsed flag --collector.systemd.unit-exclude" flag=.+\.(device|mount|scope|socket)$
ts=2023-04-05T14:40:24.356Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(sys|proc|dev|srv/somedir)($|/)
ts=2023-04-05T14:40:24.356Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tmpfs|tracefs)$
ts=2023-04-05T14:40:24.357Z caller=diskstats_common.go:99 level=warn collector=diskstats msg="--collector.diskstats.ignored-devices is DEPRECATED and will be removed in 2.0.0, use --collector.diskstats.device-exclude"
ts=2023-04-05T14:40:24.357Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(dm-|mapper/opened:|ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:110 level=info msg="Enabled collectors"
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=arp
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=btrfs
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=conntrack
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=cpu
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=cpufreq
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=diskstats
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=dmi
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=edac
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=entropy
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=filefd
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=filesystem
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=hwmon
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=loadavg
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=mdadm
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=meminfo
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=netclass
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=netdev
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=netstat
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=nfsd
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=os
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=powersupplyclass
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=schedstat
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=selinux
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=slabinfo
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=sockstat
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=softnet
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=stat
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=systemd
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=textfile
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=thermal_zone
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=time
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=timex
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=udp_queues
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=uname
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=vmstat
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=xfs
ts=2023-04-05T14:40:24.357Z caller=node_exporter.go:117 level=info collector=zfs
ts=2023-04-05T14:40:24.358Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100
ts=2023-04-05T14:40:24.358Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100

Are you running node_exporter in Docker?

no

What did you do that produced an error?

Enable the ZFS collector

What did you expect to see?

To see all arcstats.

What did you see instead?

All arcstats except for one: memory_available_bytes (as node_zfs_arc_memory_available_bytes).

It's right there in my procfile /proc/spl/kstat/zfs/arcstats:

[...]
memory_all_bytes                4    270106497024
memory_free_bytes               4    24882790400
memory_available_bytes          3    15943038848
arc_no_grow                     4    0
arc_tempreserve                 4    0
[...]

I think it is omitted by the collector because it's filtered out erroneously in this line:

node_exporter/collector/zfs_linux.go

Line 166 in b87c6a8

if parts[1] == kstatDataUint64 {

memory_available_bytes is not of type=4, but type=3 (signed instead of unsigned, I think), so it's not considered a valid metric by node_exporter. :-(

Please fix the collector so that it supports more than just the uint64 type. Thanks.

[SuperQ]

Odd, these metrics seem to be duplicates of the memstat metrics.

memory_all_bytes                4    33506664448
memory_free_bytes               4    5882482688
memory_available_bytes          3    4659681024

Compared to /proc/meminfo

MemTotal:       32721352 kB
MemFree:         5357628 kB
MemAvailable:    5531272 kB

MemTotal matches, but the others don't.

[gertvdijk]

IIUC, it's a totally different metric; /proc/meminfo is what the kernel reports as available, arcstats's memory_available_bytes is what's available within the configured max ARC and used for growing/shrinking it.

They also don't line up at all for me:

$ grep -E '(memory_available_bytes|MemAvailable)' /proc/spl/kstat/zfs/arcstats /proc/meminfo
/proc/spl/kstat/zfs/arcstats:memory_available_bytes          3    16033675136
/proc/meminfo:MemAvailable:   22941052 kB

So please keep them. I found this issue because I'm debugging an issue with the ZFS ARC for which I need that metric.

[dongjiang1989]

IIUC, it's a totally different metric; /proc/meminfo is what the kernel reports as available, arcstats's memory_available_bytes is what's available within the configured max ARC and used for growing/shrinking it.

They also don't line up at all for me:

$ grep -E '(memory_available_bytes|MemAvailable)' /proc/spl/kstat/zfs/arcstats /proc/meminfo
/proc/spl/kstat/zfs/arcstats:memory_available_bytes          3    16033675136
/proc/meminfo:MemAvailable:   22941052 kB

So please keep them. I found this issue because I'm debugging an issue with the ZFS ARC for which I need that metric.

I face the same problem