Add /proc/net/rpc stats

[wrouesnel]

Just had some interesting times diagnosing a problem where this data would prove helpful to alert off of in the future. This issue is a note to myself to implement it.

[f0]

@wrouesnel what is in the /proc/net/rpc file? Does not exist on my system.

[wrouesnel]

It's NFS operation stats. Shows counters for requests.

[f0]

@wrouesnel thx

[matthiasr]

Oh yeah I'd like that 👍

[acdha]

In addition to this, parsing /proc/self/mountstats would allow you to get per-mountpoint stats with some useful data (operation rate, RPC backlog, retransmits, average round-trip and execution times, etc.) and counts of each unique operation. The catch is that the format wasn't designed to be easy to parse as can be seen by http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=blob_plain;f=tools/mountstats/mountstats.py;hb=refs/heads/master

[SuperQ]

How much overlap is there between /proc/net/rpc/ and /proc/self/mountstats? It seems like on atleast 3.2 the rpc directory has a number of stats, and they seem to be in a more machine-readable format.

[acdha]

@SuperQ It's trading hassle for granularity. /proc/net/rpc/nfs reports global stats for the entire NFS client while /proc/self/mountstats includes more details and breaks it down for each mount but the format wasn't designed to be easily parsed. I work in an environment with a bunch of different NFS servers and mounts so the ability to break the metrics down by mount would be really useful but I recognize that it's a lot more work.

Here's an example with a 4.6-series kernel:

/proc/net/rpc/nfs:

net 0 0 0 0
rpc 105 0 105
proc3 22 0 6 0 86 3 0 2 0 0 0 0 0 0 0 0 0 0 3 0 2 1 0
proc4 59 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

/proc/self/mountstats:

device sysfs mounted on /sys with fstype sysfs
device proc mounted on /proc with fstype proc
device devtmpfs mounted on /dev with fstype devtmpfs
…
device server.example.org:/path/to/export mounted on /mnt/test with fstype nfs statvers=1.1
    opts:   rw,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=1.2.3.4,mountvers=3,mountport=4046,mountproto=udp,local_lock=none
    age:    78
    caps:   caps=0x3fc7,wtmult=4096,dtsize=4096,bsize=0,namlen=255
    sec:    flavor=1,pseudoflavor=1
    events: 6 306 0 0 7 86 395 0 0 2 0 14 2 0 4 0 0 6 0 0 0 0 0 0 0 0 0 
    bytes:  143698 0 0 0 23907 0 6 0 
    RPC iostats version: 1.0  p/v: 100003/3 (nfs)
    xprt:   tcp 812 1 1 0 65 105 105 0 105 0 2 0 0
    per-op statistics
            NULL: 0 0 0 0 0 0 0 0
         GETATTR: 6 6 0 840 672 0 4 4
         SETATTR: 0 0 0 0 0 0 0 0
          LOOKUP: 86 86 0 13424 10120 0 75 78
          ACCESS: 3 3 0 432 360 0 6 6
        READLINK: 0 0 0 0 0 0 0 0
            READ: 2 2 0 304 24164 0 18 18
           WRITE: 0 0 0 0 0 0 0 0
          CREATE: 0 0 0 0 0 0 0 0
           MKDIR: 0 0 0 0 0 0 0 0
         SYMLINK: 0 0 0 0 0 0 0 0
           MKNOD: 0 0 0 0 0 0 0 0
          REMOVE: 0 0 0 0 0 0 0 0
           RMDIR: 0 0 0 0 0 0 0 0
          RENAME: 0 0 0 0 0 0 0 0
            LINK: 0 0 0 0 0 0 0 0
         READDIR: 0 0 0 0 0 0 0 0
     READDIRPLUS: 3 3 0 492 8028 0 12 12
          FSSTAT: 0 0 0 0 0 0 0 0
          FSINFO: 2 2 0 272 328 0 1 1
        PATHCONF: 1 1 0 136 140 0 0 0
          COMMIT: 0 0 0 0 0 0 0 0

[SuperQ]

Thanks for the summary. I agree, more hassle. I think the per-mount and other details are worth it. We already parse some fairly annoying outputs like /proc/mdstat.

It would be nice to reference some kernel docs on what all of the fields mean.

[acdha]

It definitely seems to be under-documented. proc(5) has the basic structure but there's not a ton of detail (e.g. http://linux.die.net/man/5/proc) so I've been referring to the mountstats.py file in http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=blob;f=tools/mountstats/mountstats.py;hb=HEAD

[acdha]

It looks like most use of /proc/self/mountstats is through the mountstats.py file in nfs-utils, which is somewhat unfortunately hard to work with. I tossed together a quick proof of concept in a fork which modifies the mountstats Python code to make it easier to get structured values out and has a rudimentary hack to generate something plausibly compatible with textfile:

https://github.com/acdha/nfs-utils/compare/machine-readable-output

With that, I get something like this output:

https://gist.github.com/acdha/57825d3c99e5dd60acbfa7f71bc09c2c

Currently thinking that each operation should be a label rather than a separate metric but otherwise wondering what might be worth changing before porting it into Go.

[SuperQ]

Just coming back around to thinking about this one. Anyone volunteers to implement this parser in go? :-D

[mdlayher]

I'd be happy to take a shot at this.

[mdlayher]

To clarify, is this issue for tracking NFS client, server, or both statistics?

[SuperQ]

I think we are interested in adding /proc/self/mountstats for tracking NFS client stats first.

[mdlayher]

Cool. I'll look into writing a general purpose mountstats package, and once I've made some progress, I'll make a PR to vendor it in and read out some data for Prometheus metrics.