Zfs collector looking for dbuf_stats instead of dbufstats

[chr0mag]

Host operating system: output of uname -a

Linux ryzen 6.1.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 21 Dec 2022 22:27:55 +0000 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.4.0 (branch: tarball, revision: 1.4.0)
  build user:       someone@builder
  build date:       
  go version:       go1.19.1
  platform:         linux/amd64

node_exporter command line flags

/usr/bin/prometheus-node-exporter --web.config=/etc/prometheus/web-config.yml --collector.systemd --collector.processes --log.level=debug

node_exporter log output (grep zfs)

Dec 30 14:35:32 ryzen prometheus-node-exporter[3525677]: ts=2022-12-30T22:35:32.365Z caller=zfs_linux.go:53 level=debug collector=zfs msg="Cannot open file for reading" path=/proc/spl/kstat/zfs/xuio_stats
Dec 30 14:35:32 ryzen prometheus-node-exporter[3525677]: ts=2022-12-30T22:35:32.365Z caller=zfs.go:81 level=debug collector=zfs err="ZFS / ZFS statistics are not available"
Dec 30 14:35:32 ryzen prometheus-node-exporter[3525677]: ts=2022-12-30T22:35:32.367Z caller=zfs_linux.go:53 level=debug collector=zfs msg="Cannot open file for reading" path=/proc/spl/kstat/zfs/dbuf_stats
Dec 30 14:35:32 ryzen prometheus-node-exporter[3525677]: ts=2022-12-30T22:35:32.367Z caller=zfs.go:81 level=debug collector=zfs err="ZFS / ZFS statistics are not available"
Dec 30 14:35:32 ryzen prometheus-node-exporter[3525677]: ts=2022-12-30T22:35:32.368Z caller=collector.go:173 level=debug msg="collector succeeded" name=zfs duration_seconds=0.002515521

Are you running node_exporter in Docker?

no

What did you do that produced an error?

Ran the node exporter on a host using OpenZFS 2.1.7

What did you expect to see?

Expected dbuf stats to get collected.

% ls -l /proc/spl/kstat/zfs/dbufs*
-rw------- 1 root root 0 Dec 23 13:08 /proc/spl/kstat/zfs/dbufs
-rw-r--r-- 1 root root 0 Dec 23 13:08 /proc/spl/kstat/zfs/dbufstats

What did you see instead?

A debug log message stating the zfs collector was unable to find path=/proc/spl/kstat/zfs/dbuf_stats . As seen above I have both a dbufs and dbufstats file under /proc/spl/kstat/zfs/ but no dbuf_stats.

Is the ZFS collector just looking for the wrong filename?

This is the contents of dbufstats:

% cat /proc/spl/kstat/zfs/dbufstats
14 1 0x01 43 11696 36242704790 690680699874674
name                            type data
cache_count                     4    48703
cache_size_bytes                4    946789376
cache_size_bytes_max            4    1084386816
cache_target_bytes              4    1052340096
cache_lowater_bytes             4    947106087
cache_hiwater_bytes             4    1157574105
cache_total_evicts              4    1871336
cache_level_0                   4    48690
cache_level_1                   4    7
cache_level_2                   4    0
cache_level_3                   4    0
cache_level_4                   4    0
cache_level_5                   4    6
cache_level_6                   4    0
cache_level_7                   4    0
cache_level_8                   4    0
cache_level_9                   4    0
cache_level_10                  4    0
cache_level_11                  4    0
cache_level_0_bytes             4    945085440
cache_level_1_bytes             4    917504
cache_level_2_bytes             4    0
cache_level_3_bytes             4    0
cache_level_4_bytes             4    0
cache_level_5_bytes             4    786432
cache_level_6_bytes             4    0
cache_level_7_bytes             4    0
cache_level_8_bytes             4    0
cache_level_9_bytes             4    0
cache_level_10_bytes            4    0
cache_level_11_bytes            4    0
hash_hits                       4    855727905
hash_misses                     4    23824932
hash_collisions                 4    1363943
hash_elements                   4    78644
hash_elements_max               4    87544
hash_chains                     4    2839
hash_chain_max                  4    4
hash_insert_race                4    352
metadata_cache_count            4    2704
metadata_cache_size_bytes       4    6020608
metadata_cache_size_bytes_max   4    6059520
metadata_cache_overflow         4    0

[discordianfish]

Hrm maybe this was renamed in a openzfs/kernel version? Is the content the same?

[chr0mag]

Hrm maybe this was renamed in a openzfs/kernel version?

That was my first thought too, but this looks like where the filename is created and it doesn't appear to have changed in 5 years.

The zfs collector path hasn't changed in 6 years but it seems unlikely that dbufstats collection has simply been broken for years.

Is the content the same?

I don't have a system with /proc/spl/kstat/zfs/dbuf_stats so can't say.

[discordianfish]

Hrm same for me:

$ ls /proc/spl/kstat/zfs/dbuf*
/proc/spl/kstat/zfs/dbufs  /proc/spl/kstat/zfs/dbufstats

No idea what is going one tbh. Still seems like this was renamed. I suggest we just look for both files.

[astappiev]

Same issue on Debian 11

[discordianfish]

Yeah lets just look for both files. If anyone want to open a PR, should be trivial.