node_exporter hangs in a statfs syscall

[MarkusTeufelberger]

Host operating system: output of uname -a

Linux xxxXXXxxx 4.13.0-37-generic #42~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 0.15.2 (branch: HEAD, revision: 98bc64930d34878b84a0f87dfe6e1a6da61e532d)
  build user:       root@d5c4792c921f
  build date:       20171205-14:50:53
  go version:       go1.9.2

node_exporter command line flags

--web.listen-address=127.0.0.1:9100 '--collector.diskstats.ignored-devices=^(ram|loop|fd|rpc_pipefs)\d*$' --collector.diskstats --collector.filesystem --collector.loadavg --collector.meminfo --collector.stat --collector.textfile --collector.time --collector.netdev --collector.netstat --collector.filefd --collector.vmstat --collector.sockstat --collector.tcpstat --collector.entropy --collector.systemd --collector.ntp --collector.systemd.private --collector.textfile.directory=/etc/prometheus-node-exporter/textfile-collector

Are you running node_exporter in Docker?

No.

What did you do that produced an error?

Apparently there were issues with NFS causing some mounts to become unresponsive

What did you expect to see?

All metrics except the NFS ones are still exported.

What did you see instead?

node_exporter hangs in a statfs syscall.

(2414 is the PID of node-exporter)

root@xxxXXXxxx ~ # cat /proc/2414/stack                                                                                                                                                                           
[<ffffffffc048bb54>] rpc_wait_bit_killable+0x24/0xa0 [sunrpc]                                                                                                                                                       
[<ffffffffc048ca0e>] __rpc_execute+0x15e/0x420 [sunrpc]                                                                                                                                                             
[<ffffffffc048ef01>] rpc_execute+0x61/0xc0 [sunrpc]                                                                                                                                                                 
[<ffffffffc047fe48>] rpc_run_task+0xf8/0x140 [sunrpc]                                                                                                                                                               
[<ffffffffc0482020>] rpc_call_sync+0x50/0x90 [sunrpc]                                                                                                                                                               
[<ffffffffc08c5729>] nfs3_rpc_wrapper.constprop.12+0x39/0xa0 [nfsv3]                                                                                                                                                
[<ffffffffc08c63d6>] nfs3_proc_getattr+0x56/0xb0 [nfsv3]                                                                                                                                                            
[<ffffffffc0893270>] __nfs_revalidate_inode+0xd0/0x360 [nfs]                                                                                                                                                        
[<ffffffffc088a785>] nfs_weak_revalidate+0xb5/0x110 [nfs]                                                                                                                                                           
[<ffffffffa746271c>] complete_walk+0x6c/0x80                                                                                                                                                                        
[<ffffffffa7464650>] path_lookupat+0x110/0x230                                                                                                                                                                      
[<ffffffffa7467948>] filename_lookup+0xb8/0x1a0                                                                                                                                                                     
[<ffffffffa7467b06>] user_path_at_empty+0x36/0x40                                                                                                                                                                   
[<ffffffffa748fce6>] user_statfs+0x46/0xa0                                                                                                                                                                          
[<ffffffffa748fd67>] SYSC_statfs+0x27/0x60                                                                                                                                                                          
[<ffffffffa74900ee>] SyS_statfs+0xe/0x10                                                                                                                                                                            
[<ffffffffa7c001a1>] entry_SYSCALL_64_fastpath+0x24/0xab                                                                                                                                                            
[<ffffffffffffffff>] 0xffffffffffffffff  

[SuperQ]

This is a duplicate of #244, basically there is nothing we can do to prevent hangs in syscalls, as they are out of our control.

We typically recommend excluding NFS (and other network filesystems) from the metrics, and monitoring those at the source of the filesystem (NFS server)

The other difficulty with just excluding metrics for filesystems we can't scan is that this now silently drops metrics that are expected. This causes bad monitoring behavior, so we prefer an "all or nothing" approach to exposing metrics in order to make sure the operator is notified that "something is wrong".

[MarkusTeufelberger]

I don't want my car to immediately shut down all components and engage the breaks just because the radio is not receiving music at the moment.

Similarly, just because a single metric is stuck, doesn't mean I am suddenly no longer interested in all other metrics... if anything, especially in this situation I need the remaining metrics to be available at this very moment to find out which "something" is wrong. It would be a bad monitoring practice to keep this situation permanently, but I consider it bad metric collection practice to stop exposing any metrics especially at times where some error is happening.

Back on topic:

The NFS server is not under my control, so I can't collect metrics there. Just not collecting metrics for NFS is also not really an option... Is it recommended to run 2 node_exporters (one for everything but file systems, one only for file systems) or how in practice should one reliably get metrics about file systems with node_exporter?

[SuperQ]

But you do want the car to shut down if the steering wheel is stuck. We consider all metrics to be important for monitoring. We consider metrics vanishing and re-appearing unexpectedly to be very bad behavior for exporters. Like I said above, the all-or-nothing is what we intend to happen.

Monitoring an NFS server outside your control is a difficult situation. We provide NFS client RPC metrics that don't block on the filesystem. For monitoring the storage space, I might consider making a textfile collector script. You would need to name the metric differently, so it would need to be something like node_nfs_available_bytes or similar.

I wish there was a better solution, but POSIX syscalls don't make this easy. ☹️

[MarkusTeufelberger]

No to take the analogy too far, but I don't want the car windows to turn blank if the steering wheel is stuck so I don't have any way of knowing which state the system is in... Especially in this situation I want my GPS to tell me I'm still in my garage and not on the highway instead of just telling me nothing at all (not even logging which metric is stuck!).

If I understand you correctly, you are suggesting that I automatically run df and collect the output of that as a text file because node_exporter can't do this for me?!

I agree that having syscalls that put the calling thread in uninterruptible sleep potentially indefinitely with no way to cancel or timeout sucks. There are solutions to this problem though, I'm sure. Also, if this is not recommended and discussed since 2 years in a bug report and you don't want to introduce state to the metrics collection, why is the broken/problematic nfs module still enabled by default instead of having a documented alternative with systemd timers, text exporters and nonblocking RPC metrics?

[grobie]

There are solutions to this problem though, I'm sure.

The first one would be to mount NFS without allowing it to stall your system. There are multiple options for this if I remember correctly, but setting the intr mount option should help already https://www.centos.org/docs/5/html/Deployment_Guide-en-US/s1-nfs-client-config-options.html

[MarkusTeufelberger]

https://linux.die.net/man/5/nfs

The intr / nointr mount option is deprecated after kernel 2.6.25. Only SIGKILL can interrupt a pending NFS operation on these kernels, and if specified, this mount option is ignored to provide backwards compatibility with older kernels.

[grobie]

Thanks for the info. In that case, I think it's reasonable to disable the nfs collector by default, document these issues and let users explicitely opt-in.

[MarkusTeufelberger]

The proper solution would be to implement exponential backoff and also export meta-metrics about hanging collectors...

[grobie]

@MarkusTeufelberger PRs are welcome.

[MarkusTeufelberger]

Even ones that introduce state to node_exporter? This was the reason #244 was in essence dismissed as impossible with current design goals...

[discordianfish]

@MarkusTeufelberger That depends. The exporter shouldn't be suddenly start or stop scraping particular filesystems but if you have an idea how to make it timeout on statfs syscalls or collectors in general (and expose meta-metrics about this), we're happy to consider this.
Either way, let's close this as dup of #244 and discuss over there.

[SuperQ]

@grobie, this isn't about the nfs collector, that is fine, non-blocking. This is filesystem collector interacting with nfs.