NFS server's disconnection caused the host high load and high VIRT size

[wfhu]

Host operating system: output of uname -a

Linux 3.10.0-862.6.3.el7.x86_64 #1 SMP Tue Jun 26 16:32:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 0.16.0 (branch: HEAD, revision: d42bd70)
build user: root@a67a9bc13a69
build date: 20180515-15:52:42
go version: go1.9.6

node_exporter command line flags

/usr/local/prometheus/node_exporter-0.16.0.linux-amd64/node_exporter

Are you running node_exporter in Docker?

No, not in Docker

What did you do that produced an error?

The server has mounted a NFS volume to local directory, when the NFS server is down, the server's load average increase to 330.
After I restarted the node_exporter process, everything came back to normal.

What did you expect to see?

This should not happen, the node_exporter should detect that the NFS server is down.

What did you see instead?

top - 16:41:00 up 103 days, 20:04, 2 users, load average: 331.40, 330.96, 302.72
Tasks: 222 total, 1 running, 221 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.4 us, 1.1 sy, 0.0 ni, 96.8 id, 0.7 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 65975208 total, 487048 free, 33177676 used, 32310484 buff/cache
KiB Swap: 16777212 total, 16654076 free, 123136 used. 31770544 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19263 root 20 0 0 0 0 D 0.0 0.0 0:00.00 172.16.89.197-m
21651 nobody 20 0 10.924g 200524 5236 D 0.0 0.3 0:25.59 node_exporter

As we can see from the above, the VIRT of node_exporter is about 10G, and the load of the server rised up to 331, which are VERY high.

After I kill -9 the node_exporter process, the load drop quickly.

top - 16:41:06 up 103 days, 20:04, 2 users, load average: 305.02, 325.50, 301.11
Tasks: 223 total, 2 running, 221 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.2 us, 1.2 sy, 0.0 ni, 94.8 id, 1.8 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 65975208 total, 919604 free, 32944664 used, 32110940 buff/cache
KiB Swap: 16777212 total, 16654076 free, 123136 used. 32004128 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19263 root 20 0 0 0 0 D 0.0 0.0 0:00.00 172.16.89.197-m

after just 4 minutes, everything is normal now

top -b -n 1 | awk '{if (NR <=7) print; else if ($8 == "D") {print;
count++} } END {print "Total status D: "count}'

top - 16:45:40 up 103 days, 20:09, 2 users, load average: 5.06, 132.79, 225.54
Tasks: 221 total, 1 running, 220 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.4 us, 1.2 sy, 0.0 ni, 95.4 id, 1.9 wa, 0.0 hi, 0.0 si, 0.1 st
KiB Mem : 65975208 total, 841576 free, 32951132 used, 32182500 buff/cache
KiB Swap: 16777212 total, 16654076 free, 123136 used. 31997280 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19263 root 20 0 0 0 0 D 0.0 0.0 0:00.00 172.16.89.197-m
Total status D: 1

and the node_exporter's VIRT size:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
nobody 25360 0.4 0.0 1263600 12488 ? Ssl 16:41 0:46 /usr/local/prometheus/node_exporter-0.16.0.linux-amd64/node_exporter

[wfhu]

Add the output of command lsof of the node_exporter process before it was restarted, at this moment, the non-working NFS directory was umounted already:

[root@nfjd log]# top -b -n 1 | awk '{if (NR <=7) print; else if ($8 == "D") {print;
count++} } END {print "Total status D: "count}'
top - 16:39:44 up 103 days, 20:03, 2 users, load average: 331.38, 330.84, 300.15
Tasks: 221 total, 1 running, 220 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.6 us, 0.4 sy, 0.0 ni, 95.7 id, 3.3 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 65975208 total, 509720 free, 33175900 used, 32289588 buff/cache
KiB Swap: 16777212 total, 16654076 free, 123136 used. 31771988 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19263 root 20 0 0 0 0 D 0.0 0.0 0:00.00 172.16.89.197-m
21651 nobody 20 0 10.916g 200524 5236 D 0.0 0.3 0:24.57 node_exporter
Total status D: 2
[root@nfjd log]# lsof -p 21651
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
node_expo 21651 nobody cwd DIR 253,2 267 64 /
node_expo 21651 nobody rtd DIR 253,2 267 64 /
node_expo 21651 nobody txt REG 253,2 16900416 320929514 /usr/local/prometheus/node_exporter-0.16.0.linux-amd64/node_exporter
node_expo 21651 nobody 0r CHR 1,3 0t0 1028 /dev/null
node_expo 21651 nobody 1u unix 0xffff9d4fba2bb800 0t0 25185665 socket
node_expo 21651 nobody 2u unix 0xffff9d4fba2bb800 0t0 25185665 socket
node_expo 21651 nobody 3u IPv6 25179422 0t0 TCP *:jetdirect (LISTEN)
node_expo 21651 nobody 4u a_inode 0,10 0 6231 [eventpoll]
node_expo 21651 nobody 5u sock 0,7 0t0 25184760 protocol: TCPv6
node_expo 21651 nobody 6u sock 0,7 0t0 25170719 protocol: TCPv6
node_expo 21651 nobody 7u sock 0,7 0t0 25184763 protocol: TCPv6
node_expo 21651 nobody 8u sock 0,7 0t0 25181863 protocol: TCPv6
node_expo 21651 nobody 9u sock 0,7 0t0 25185679 protocol: TCPv6
node_expo 21651 nobody 10u sock 0,7 0t0 25181865 protocol: TCPv6
node_expo 21651 nobody 11u sock 0,7 0t0 25181867 protocol: TCPv6
node_expo 21651 nobody 12u sock 0,7 0t0 25184777 protocol: TCPv6
node_expo 21651 nobody 13u sock 0,7 0t0 25185683 protocol: TCPv6
node_expo 21651 nobody 14u sock 0,7 0t0 25181883 protocol: TCPv6
node_expo 21651 nobody 15u sock 0,7 0t0 25184789 protocol: TCPv6
node_expo 21651 nobody 16u sock 0,7 0t0 25181885 protocol: TCPv6
node_expo 21651 nobody 17u sock 0,7 0t0 25184795 protocol: TCPv6
node_expo 21651 nobody 18u sock 0,7 0t0 25184797 protocol: TCPv6
node_expo 21651 nobody 19u sock 0,7 0t0 25183598 protocol: TCPv6
node_expo 21651 nobody 20u sock 0,7 0t0 25170741 protocol: TCPv6
node_expo 21651 nobody 21u sock 0,7 0t0 25183600 protocol: TCPv6
node_expo 21651 nobody 22u sock 0,7 0t0 25177622 protocol: TCPv6
node_expo 21651 nobody 23u sock 0,7 0t0 25183609 protocol: TCPv6
node_expo 21651 nobody 24u sock 0,7 0t0 25181901 protocol: TCPv6
node_expo 21651 nobody 25u sock 0,7 0t0 25184804 protocol: TCPv6
node_expo 21651 nobody 26u sock 0,7 0t0 25181903 protocol: TCPv6
node_expo 21651 nobody 27u sock 0,7 0t0 25185704 protocol: TCPv6
node_expo 21651 nobody 28u sock 0,7 0t0 25185706 protocol: TCPv6
node_expo 21651 nobody 29u sock 0,7 0t0 25184810 protocol: TCPv6
node_expo 21651 nobody 30u sock 0,7 0t0 25183612 protocol: TCPv6
node_expo 21651 nobody 31u sock 0,7 0t0 25177631 protocol: TCPv6
node_expo 21651 nobody 32u sock 0,7 0t0 25183626 protocol: TCPv6
node_expo 21651 nobody 33u sock 0,7 0t0 25181919 protocol: TCPv6
node_expo 21651 nobody 34u sock 0,7 0t0 25184825 protocol: TCPv6
node_expo 21651 nobody 35u sock 0,7 0t0 25184830 protocol: TCPv6
node_expo 21651 nobody 36u sock 0,7 0t0 25181958 protocol: TCPv6
node_expo 21651 nobody 37u sock 0,7 0t0 25181961 protocol: TCPv6
node_expo 21651 nobody 38u sock 0,7 0t0 25176361 protocol: TCPv6
node_expo 21651 nobody 39u sock 0,7 0t0 25184834 protocol: TCPv6
node_expo 21651 nobody 40u sock 0,7 0t0 25183659 protocol: TCPv6
node_expo 21651 nobody 41u sock 0,7 0t0 25181973 protocol: TCPv6
node_expo 21651 nobody 42u sock 0,7 0t0 25181975 protocol: TCPv6
node_expo 21651 nobody 43u sock 0,7 0t0 25181978 protocol: TCPv6
node_expo 21651 nobody 44u sock 0,7 0t0 25181980 protocol: TCPv6
node_expo 21651 nobody 45u sock 0,7 0t0 25184859 protocol: TCPv6
node_expo 21651 nobody 46u sock 0,7 0t0 25181987 protocol: TCPv6
node_expo 21651 nobody 47u sock 0,7 0t0 25162289 protocol: TCPv6
node_expo 21651 nobody 48u sock 0,7 0t0 25185766 protocol: TCPv6
node_expo 21651 nobody 49u sock 0,7 0t0 25182600 protocol: TCPv6
node_expo 21651 nobody 50u sock 0,7 0t0 25183697 protocol: TCPv6
node_expo 21651 nobody 51u sock 0,7 0t0 25185783 protocol: TCPv6
node_expo 21651 nobody 52u sock 0,7 0t0 25183701 protocol: TCPv6
node_expo 21651 nobody 53u sock 0,7 0t0 25183704 protocol: TCPv6
node_expo 21651 nobody 54u sock 0,7 0t0 25182008 protocol: TCPv6
node_expo 21651 nobody 55u sock 0,7 0t0 25183706 protocol: TCPv6
node_expo 21651 nobody 56u sock 0,7 0t0 25183708 protocol: TCPv6
node_expo 21651 nobody 57u sock 0,7 0t0 25184868 protocol: TCPv6
node_expo 21651 nobody 58u sock 0,7 0t0 25182018 protocol: TCPv6
node_expo 21651 nobody 59u sock 0,7 0t0 25182024 protocol: TCPv6
node_expo 21651 nobody 60u sock 0,7 0t0 25185809 protocol: TCPv6
node_expo 21651 nobody 61u sock 0,7 0t0 25184877 protocol: TCPv6
node_expo 21651 nobody 62u sock 0,7 0t0 25186317 protocol: TCPv6
node_expo 21651 nobody 63u sock 0,7 0t0 25182032 protocol: TCPv6
node_expo 21651 nobody 64u sock 0,7 0t0 25184884 protocol: TCPv6
node_expo 21651 nobody 65u sock 0,7 0t0 25185811 protocol: TCPv6
node_expo 21651 nobody 66u sock 0,7 0t0 25184890 protocol: TCPv6
node_expo 21651 nobody 67u sock 0,7 0t0 25183726 protocol: TCPv6
node_expo 21651 nobody 68u sock 0,7 0t0 25182053 protocol: TCPv6
node_expo 21651 nobody 69u sock 0,7 0t0 25183730 protocol: TCPv6
node_expo 21651 nobody 70u sock 0,7 0t0 25183734 protocol: TCPv6
node_expo 21651 nobody 71u sock 0,7 0t0 25184906 protocol: TCPv6
node_expo 21651 nobody 72u sock 0,7 0t0 25185823 protocol: TCPv6
node_expo 21651 nobody 73u sock 0,7 0t0 25183736 protocol: TCPv6
node_expo 21651 nobody 74u sock 0,7 0t0 25185825 protocol: TCPv6
node_expo 21651 nobody 75u sock 0,7 0t0 25183738 protocol: TCPv6
node_expo 21651 nobody 76u sock 0,7 0t0 25184911 protocol: TCPv6
node_expo 21651 nobody 77u sock 0,7 0t0 25184918 protocol: TCPv6
node_expo 21651 nobody 78u sock 0,7 0t0 25185833 protocol: TCPv6
node_expo 21651 nobody 79u sock 0,7 0t0 25183748 protocol: TCPv6
node_expo 21651 nobody 80u sock 0,7 0t0 25182081 protocol: TCPv6
node_expo 21651 nobody 81u sock 0,7 0t0 25182092 protocol: TCPv6
node_expo 21651 nobody 82u sock 0,7 0t0 25185843 protocol: TCPv6
node_expo 21651 nobody 83u sock 0,7 0t0 25184931 protocol: TCPv6
node_expo 21651 nobody 84u sock 0,7 0t0 25183783 protocol: TCPv6
node_expo 21651 nobody 85u sock 0,7 0t0 25185851 protocol: TCPv6
node_expo 21651 nobody 86u sock 0,7 0t0 25183792 protocol: TCPv6
node_expo 21651 nobody 87u sock 0,7 0t0 25182109 protocol: TCPv6
node_expo 21651 nobody 88u sock 0,7 0t0 25185858 protocol: TCPv6
node_expo 21651 nobody 89u sock 0,7 0t0 25185865 protocol: TCPv6
node_expo 21651 nobody 90u sock 0,7 0t0 25184945 protocol: TCPv6
node_expo 21651 nobody 91u sock 0,7 0t0 25183803 protocol: TCPv6
node_expo 21651 nobody 92u sock 0,7 0t0 25185867 protocol: TCPv6
node_expo 21651 nobody 93u sock 0,7 0t0 25182115 protocol: TCPv6
node_expo 21651 nobody 94u sock 0,7 0t0 25183809 protocol: TCPv6
node_expo 21651 nobody 95u sock 0,7 0t0 25183815 protocol: TCPv6
node_expo 21651 nobody 96u sock 0,7 0t0 25183817 protocol: TCPv6
node_expo 21651 nobody 97u sock 0,7 0t0 25184953 protocol: TCPv6
node_expo 21651 nobody 98u sock 0,7 0t0 25184962 protocol: TCPv6
node_expo 21651 nobody 99u sock 0,7 0t0 25185872 protocol: TCPv6
node_expo 21651 nobody 100u sock 0,7 0t0 25183821 protocol: TCPv6
node_expo 21651 nobody 101u sock 0,7 0t0 25185874 protocol: TCPv6
node_expo 21651 nobody 102u sock 0,7 0t0 25184966 protocol: TCPv6
node_expo 21651 nobody 103u sock 0,7 0t0 25182151 protocol: TCPv6
node_expo 21651 nobody 104u sock 0,7 0t0 25183834 protocol: TCPv6
node_expo 21651 nobody 105u sock 0,7 0t0 25184971 protocol: TCPv6
node_expo 21651 nobody 106u sock 0,7 0t0 25184975 protocol: TCPv6
node_expo 21651 nobody 107u sock 0,7 0t0 25183840 protocol: TCPv6
node_expo 21651 nobody 108u sock 0,7 0t0 25184978 protocol: TCPv6
node_expo 21651 nobody 109u sock 0,7 0t0 25183843 protocol: TCPv6
node_expo 21651 nobody 110u sock 0,7 0t0 25183845 protocol: TCPv6
node_expo 21651 nobody 111u sock 0,7 0t0 25185893 protocol: TCPv6
node_expo 21651 nobody 112u sock 0,7 0t0 25185896 protocol: TCPv6
node_expo 21651 nobody 113u sock 0,7 0t0 25183854 protocol: TCPv6
node_expo 21651 nobody 114u sock 0,7 0t0 25183856 protocol: TCPv6
node_expo 21651 nobody 115u sock 0,7 0t0 25184994 protocol: TCPv6
node_expo 21651 nobody 116u sock 0,7 0t0 25182179 protocol: TCPv6
node_expo 21651 nobody 117u sock 0,7 0t0 25185917 protocol: TCPv6
node_expo 21651 nobody 118u sock 0,7 0t0 25185919 protocol: TCPv6
node_expo 21651 nobody 119u sock 0,7 0t0 25184996 protocol: TCPv6
node_expo 21651 nobody 120u sock 0,7 0t0 25182187 protocol: TCPv6
node_expo 21651 nobody 121u sock 0,7 0t0 25182189 protocol: TCPv6
node_expo 21651 nobody 122u sock 0,7 0t0 25182195 protocol: TCPv6
node_expo 21651 nobody 123u sock 0,7 0t0 25185933 protocol: TCPv6
node_expo 21651 nobody 124u sock 0,7 0t0 25183872 protocol: TCPv6
node_expo 21651 nobody 125u sock 0,7 0t0 25185018 protocol: TCPv6
node_expo 21651 nobody 126u sock 0,7 0t0 25187363 protocol: TCPv6
node_expo 21651 nobody 127u sock 0,7 0t0 25185029 protocol: TCPv6
node_expo 21651 nobody 128u sock 0,7 0t0 25183886 protocol: TCPv6
node_expo 21651 nobody 129u sock 0,7 0t0 25183889 protocol: TCPv6
node_expo 21651 nobody 130u sock 0,7 0t0 25183891 protocol: TCPv6
node_expo 21651 nobody 131u sock 0,7 0t0 25183896 protocol: TCPv6
node_expo 21651 nobody 132u sock 0,7 0t0 25185944 protocol: TCPv6
node_expo 21651 nobody 133u sock 0,7 0t0 25185042 protocol: TCPv6
node_expo 21651 nobody 134u sock 0,7 0t0 25185047 protocol: TCPv6
node_expo 21651 nobody 135u sock 0,7 0t0 25183902 protocol: TCPv6
node_expo 21651 nobody 136u sock 0,7 0t0 25182708 protocol: TCPv6
node_expo 21651 nobody 137u sock 0,7 0t0 25187377 protocol: TCPv6
node_expo 21651 nobody 138u sock 0,7 0t0 25183907 protocol: TCPv6
node_expo 21651 nobody 139u sock 0,7 0t0 25183909 protocol: TCPv6
node_expo 21651 nobody 140u sock 0,7 0t0 25187390 protocol: TCPv6
node_expo 21651 nobody 141u sock 0,7 0t0 25183911 protocol: TCPv6
node_expo 21651 nobody 142u sock 0,7 0t0 25183913 protocol: TCPv6
node_expo 21651 nobody 143u sock 0,7 0t0 25187400 protocol: TCPv6
node_expo 21651 nobody 144u sock 0,7 0t0 25185988 protocol: TCPv6
node_expo 21651 nobody 145u sock 0,7 0t0 25185060 protocol: TCPv6
node_expo 21651 nobody 146u sock 0,7 0t0 25185990 protocol: TCPv6
node_expo 21651 nobody 147u sock 0,7 0t0 25185994 protocol: TCPv6
node_expo 21651 nobody 148u sock 0,7 0t0 25185997 protocol: TCPv6
node_expo 21651 nobody 149u sock 0,7 0t0 25183917 protocol: TCPv6
node_expo 21651 nobody 150u sock 0,7 0t0 25187417 protocol: TCPv6
node_expo 21651 nobody 151u sock 0,7 0t0 25186004 protocol: TCPv6
node_expo 21651 nobody 152u sock 0,7 0t0 25186014 protocol: TCPv6
node_expo 21651 nobody 153u sock 0,7 0t0 25185086 protocol: TCPv6
node_expo 21651 nobody 154u sock 0,7 0t0 25183922 protocol: TCPv6
node_expo 21651 nobody 155u sock 0,7 0t0 25185088 protocol: TCPv6
node_expo 21651 nobody 156u sock 0,7 0t0 25183926 protocol: TCPv6
node_expo 21651 nobody 157u sock 0,7 0t0 25183928 protocol: TCPv6
node_expo 21651 nobody 158u sock 0,7 0t0 25185100 protocol: TCPv6
node_expo 21651 nobody 159u sock 0,7 0t0 25187435 protocol: TCPv6
node_expo 21651 nobody 160u sock 0,7 0t0 25185104 protocol: TCPv6
node_expo 21651 nobody 161u sock 0,7 0t0 25186042 protocol: TCPv6
node_expo 21651 nobody 162u sock 0,7 0t0 25183939 protocol: TCPv6
node_expo 21651 nobody 163u sock 0,7 0t0 25185107 protocol: TCPv6
node_expo 21651 nobody 164u sock 0,7 0t0 25186045 protocol: TCPv6
node_expo 21651 nobody 165u sock 0,7 0t0 25185111 protocol: TCPv6
node_expo 21651 nobody 166u sock 0,7 0t0 25185113 protocol: TCPv6
node_expo 21651 nobody 167u sock 0,7 0t0 25187456 protocol: TCPv6
node_expo 21651 nobody 168u sock 0,7 0t0 25185123 protocol: TCPv6
node_expo 21651 nobody 169u sock 0,7 0t0 25183953 protocol: TCPv6
node_expo 21651 nobody 170u sock 0,7 0t0 25187459 protocol: TCPv6
node_expo 21651 nobody 171u sock 0,7 0t0 25183995 protocol: TCPv6
node_expo 21651 nobody 172u sock 0,7 0t0 25187475 protocol: TCPv6
node_expo 21651 nobody 173u sock 0,7 0t0 25176557 protocol: TCPv6
node_expo 21651 nobody 174u sock 0,7 0t0 25186098 protocol: TCPv6
node_expo 21651 nobody 175u sock 0,7 0t0 25184003 protocol: TCPv6
node_expo 21651 nobody 176u sock 0,7 0t0 25187504 protocol: TCPv6
node_expo 21651 nobody 177u sock 0,7 0t0 25185170 protocol: TCPv6
node_expo 21651 nobody 178u sock 0,7 0t0 25184011 protocol: TCPv6
node_expo 21651 nobody 179u sock 0,7 0t0 25185178 protocol: TCPv6
node_expo 21651 nobody 180u sock 0,7 0t0 25186121 protocol: TCPv6
node_expo 21651 nobody 181u sock 0,7 0t0 25185198 protocol: TCPv6
node_expo 21651 nobody 182u sock 0,7 0t0 25185216 protocol: TCPv6
node_expo 21651 nobody 183u sock 0,7 0t0 25187559 protocol: TCPv6
node_expo 21651 nobody 184u sock 0,7 0t0 25187563 protocol: TCPv6
node_expo 21651 nobody 185u sock 0,7 0t0 25184076 protocol: TCPv6
node_expo 21651 nobody 186u sock 0,7 0t0 25186149 protocol: TCPv6
node_expo 21651 nobody 187u sock 0,7 0t0 25184083 protocol: TCPv6
node_expo 21651 nobody 188u sock 0,7 0t0 25187571 protocol: TCPv6
node_expo 21651 nobody 189u sock 0,7 0t0 25185244 protocol: TCPv6
node_expo 21651 nobody 190u sock 0,7 0t0 25184085 protocol: TCPv6
node_expo 21651 nobody 191u sock 0,7 0t0 25187573 protocol: TCPv6
node_expo 21651 nobody 192u sock 0,7 0t0 25186158 protocol: TCPv6
node_expo 21651 nobody 193u sock 0,7 0t0 25186160 protocol: TCPv6
node_expo 21651 nobody 194u sock 0,7 0t0 25189432 protocol: TCPv6
node_expo 21651 nobody 195u sock 0,7 0t0 25185260 protocol: TCPv6
node_expo 21651 nobody 196u sock 0,7 0t0 25186163 protocol: TCPv6
node_expo 21651 nobody 197u sock 0,7 0t0 25178057 protocol: TCPv6
node_expo 21651 nobody 198u sock 0,7 0t0 25185266 protocol: TCPv6
node_expo 21651 nobody 199u sock 0,7 0t0 25185268 protocol: TCPv6
node_expo 21651 nobody 200u sock 0,7 0t0 25178059 protocol: TCPv6
node_expo 21651 nobody 201u sock 0,7 0t0 25184150 protocol: TCPv6
node_expo 21651 nobody 202u sock 0,7 0t0 25187613 protocol: TCPv6
node_expo 21651 nobody 203u sock 0,7 0t0 25184161 protocol: TCPv6
node_expo 21651 nobody 204u sock 0,7 0t0 25186185 protocol: TCPv6
node_expo 21651 nobody 205u sock 0,7 0t0 25187617 protocol: TCPv6
node_expo 21651 nobody 206u sock 0,7 0t0 25186804 protocol: TCPv6
node_expo 21651 nobody 207u sock 0,7 0t0 25187623 protocol: TCPv6
node_expo 21651 nobody 208u sock 0,7 0t0 25184169 protocol: TCPv6
node_expo 21651 nobody 209u sock 0,7 0t0 25185279 protocol: TCPv6
node_expo 21651 nobody 210u sock 0,7 0t0 25190403 protocol: TCPv6
node_expo 21651 nobody 211u sock 0,7 0t0 25187637 protocol: TCPv6
node_expo 21651 nobody 212u sock 0,7 0t0 25186202 protocol: TCPv6
node_expo 21651 nobody 213u sock 0,7 0t0 25187651 protocol: TCPv6
node_expo 21651 nobody 214u sock 0,7 0t0 25190421 protocol: TCPv6
node_expo 21651 nobody 215u sock 0,7 0t0 25186218 protocol: TCPv6
node_expo 21651 nobody 216u sock 0,7 0t0 25190425 protocol: TCPv6
node_expo 21651 nobody 217u sock 0,7 0t0 25184212 protocol: TCPv6
node_expo 21651 nobody 218u sock 0,7 0t0 25184214 protocol: TCPv6
node_expo 21651 nobody 219u sock 0,7 0t0 25187727 protocol: TCPv6
node_expo 21651 nobody 220u sock 0,7 0t0 25192456 protocol: TCPv6
node_expo 21651 nobody 221u sock 0,7 0t0 25186265 protocol: TCPv6
node_expo 21651 nobody 222u sock 0,7 0t0 25187768 protocol: TCPv6
node_expo 21651 nobody 223u sock 0,7 0t0 25192487 protocol: TCPv6
node_expo 21651 nobody 224u sock 0,7 0t0 25192509 protocol: TCPv6
node_expo 21651 nobody 225u sock 0,7 0t0 25187791 protocol: TCPv6
node_expo 21651 nobody 226u sock 0,7 0t0 25190488 protocol: TCPv6
node_expo 21651 nobody 227u sock 0,7 0t0 25162440 protocol: TCPv6
node_expo 21651 nobody 228u sock 0,7 0t0 25192545 protocol: TCPv6
node_expo 21651 nobody 229u sock 0,7 0t0 25186278 protocol: TCPv6
node_expo 21651 nobody 230u sock 0,7 0t0 25187834 protocol: TCPv6
node_expo 21651 nobody 231u sock 0,7 0t0 25190547 protocol: TCPv6
node_expo 21651 nobody 232u sock 0,7 0t0 25192565 protocol: TCPv6
node_expo 21651 nobody 233u sock 0,7 0t0 25187853 protocol: TCPv6
node_expo 21651 nobody 234u sock 0,7 0t0 25187855 protocol: TCPv6
node_expo 21651 nobody 235u sock 0,7 0t0 25187857 protocol: TCPv6
node_expo 21651 nobody 236u sock 0,7 0t0 25174658 protocol: TCPv6
node_expo 21651 nobody 237u sock 0,7 0t0 25190579 protocol: TCPv6
node_expo 21651 nobody 238u sock 0,7 0t0 25192571 protocol: TCPv6
node_expo 21651 nobody 239u sock 0,7 0t0 25187865 protocol: TCPv6
node_expo 21651 nobody 240u sock 0,7 0t0 25192577 protocol: TCPv6
node_expo 21651 nobody 241u sock 0,7 0t0 25187868 protocol: TCPv6
node_expo 21651 nobody 242u sock 0,7 0t0 25195541 protocol: TCPv6
node_expo 21651 nobody 243u sock 0,7 0t0 25190589 protocol: TCPv6
node_expo 21651 nobody 244u sock 0,7 0t0 25192587 protocol: TCPv6
node_expo 21651 nobody 245u sock 0,7 0t0 25188098 protocol: TCPv6
node_expo 21651 nobody 246u sock 0,7 0t0 25190595 protocol: TCPv6
node_expo 21651 nobody 247u sock 0,7 0t0 25195565 protocol: TCPv6
node_expo 21651 nobody 248u sock 0,7 0t0 25192615 protocol: TCPv6
node_expo 21651 nobody 249u sock 0,7 0t0 25192622 protocol: TCPv6
node_expo 21651 nobody 250u sock 0,7 0t0 25188122 protocol: TCPv6
node_expo 21651 nobody 251u sock 0,7 0t0 25195597 protocol: TCPv6
node_expo 21651 nobody 252u sock 0,7 0t0 25192624 protocol: TCPv6
node_expo 21651 nobody 253u sock 0,7 0t0 25188124 protocol: TCPv6
node_expo 21651 nobody 254u sock 0,7 0t0 25192626 protocol: TCPv6
node_expo 21651 nobody 255u sock 0,7 0t0 25195611 protocol: TCPv6
node_expo 21651 nobody 256u sock 0,7 0t0 25190619 protocol: TCPv6
node_expo 21651 nobody 257u sock 0,7 0t0 25190627 protocol: TCPv6
node_expo 21651 nobody 258u sock 0,7 0t0 25190634 protocol: TCPv6
node_expo 21651 nobody 259u sock 0,7 0t0 25195631 protocol: TCPv6
node_expo 21651 nobody 260u sock 0,7 0t0 25195641 protocol: TCPv6
node_expo 21651 nobody 261u sock 0,7 0t0 25190688 protocol: TCPv6
node_expo 21651 nobody 262u sock 0,7 0t0 25192673 protocol: TCPv6
node_expo 21651 nobody 263u sock 0,7 0t0 25192675 protocol: TCPv6
node_expo 21651 nobody 264u sock 0,7 0t0 25192682 protocol: TCPv6
node_expo 21651 nobody 265u sock 0,7 0t0 25190695 protocol: TCPv6
node_expo 21651 nobody 266u sock 0,7 0t0 25192696 protocol: TCPv6
node_expo 21651 nobody 267u sock 0,7 0t0 25195679 protocol: TCPv6
node_expo 21651 nobody 268u sock 0,7 0t0 25190703 protocol: TCPv6
node_expo 21651 nobody 269u sock 0,7 0t0 25190715 protocol: TCPv6
node_expo 21651 nobody 270u sock 0,7 0t0 25190718 protocol: TCPv6
node_expo 21651 nobody 271u sock 0,7 0t0 25195683 protocol: TCPv6
node_expo 21651 nobody 272u sock 0,7 0t0 25174773 protocol: TCPv6
node_expo 21651 nobody 273u sock 0,7 0t0 25190723 protocol: TCPv6
node_expo 21651 nobody 274u sock 0,7 0t0 25192710 protocol: TCPv6
node_expo 21651 nobody 275u sock 0,7 0t0 25192712 protocol: TCPv6
node_expo 21651 nobody 276u sock 0,7 0t0 25192714 protocol: TCPv6
node_expo 21651 nobody 277u sock 0,7 0t0 25195691 protocol: TCPv6
node_expo 21651 nobody 278u sock 0,7 0t0 25192720 protocol: TCPv6
node_expo 21651 nobody 279u sock 0,7 0t0 25192725 protocol: TCPv6
node_expo 21651 nobody 280u sock 0,7 0t0 25195694 protocol: TCPv6
node_expo 21651 nobody 281u sock 0,7 0t0 25188621 protocol: TCPv6
node_expo 21651 nobody 282u sock 0,7 0t0 25192729 protocol: TCPv6
node_expo 21651 nobody 283u sock 0,7 0t0 25190771 protocol: TCPv6
node_expo 21651 nobody 284u sock 0,7 0t0 25192763 protocol: TCPv6
node_expo 21651 nobody 285u sock 0,7 0t0 25188303 protocol: TCPv6
node_expo 21651 nobody 286u sock 0,7 0t0 25195727 protocol: TCPv6
node_expo 21651 nobody 287u sock 0,7 0t0 25188305 protocol: TCPv6
node_expo 21651 nobody 288u sock 0,7 0t0 25195735 protocol: TCPv6
node_expo 21651 nobody 289u sock 0,7 0t0 25188307 protocol: TCPv6
node_expo 21651 nobody 290u sock 0,7 0t0 25192845 protocol: TCPv6
node_expo 21651 nobody 291u sock 0,7 0t0 25195744 protocol: TCPv6
node_expo 21651 nobody 292u sock 0,7 0t0 25192852 protocol: TCPv6
node_expo 21651 nobody 293u sock 0,7 0t0 25195754 protocol: TCPv6
node_expo 21651 nobody 294u sock 0,7 0t0 25190839 protocol: TCPv6
node_expo 21651 nobody 295u sock 0,7 0t0 25192858 protocol: TCPv6
node_expo 21651 nobody 296u sock 0,7 0t0 25195756 protocol: TCPv6
node_expo 21651 nobody 297u sock 0,7 0t0 25192864 protocol: TCPv6
node_expo 21651 nobody 298u sock 0,7 0t0 25190850 protocol: TCPv6
node_expo 21651 nobody 299u sock 0,7 0t0 25192867 protocol: TCPv6
node_expo 21651 nobody 300u sock 0,7 0t0 25192870 protocol: TCPv6
node_expo 21651 nobody 301u sock 0,7 0t0 25192875 protocol: TCPv6
node_expo 21651 nobody 302u sock 0,7 0t0 25197800 protocol: TCPv6
node_expo 21651 nobody 303u sock 0,7 0t0 25192899 protocol: TCPv6
node_expo 21651 nobody 304u sock 0,7 0t0 25197809 protocol: TCPv6
node_expo 21651 nobody 305u sock 0,7 0t0 25197811 protocol: TCPv6
node_expo 21651 nobody 306u sock 0,7 0t0 25190896 protocol: TCPv6
node_expo 21651 nobody 307u sock 0,7 0t0 25192931 protocol: TCPv6
node_expo 21651 nobody 308u sock 0,7 0t0 25192933 protocol: TCPv6
node_expo 21651 nobody 309u sock 0,7 0t0 25192937 protocol: TCPv6
node_expo 21651 nobody 310u sock 0,7 0t0 25187012 protocol: TCPv6
node_expo 21651 nobody 311u sock 0,7 0t0 25192957 protocol: TCPv6
node_expo 21651 nobody 312u sock 0,7 0t0 25195850 protocol: TCPv6
node_expo 21651 nobody 313u sock 0,7 0t0 25192962 protocol: TCPv6
node_expo 21651 nobody 314u sock 0,7 0t0 25195864 protocol: TCPv6
node_expo 21651 nobody 315u sock 0,7 0t0 25192965 protocol: TCPv6
node_expo 21651 nobody 316u sock 0,7 0t0 25192967 protocol: TCPv6
node_expo 21651 nobody 317u sock 0,7 0t0 25195866 protocol: TCPv6
node_expo 21651 nobody 318u sock 0,7 0t0 25195869 protocol: TCPv6
node_expo 21651 nobody 319u sock 0,7 0t0 25190964 protocol: TCPv6
node_expo 21651 nobody 320u sock 0,7 0t0 25197881 protocol: TCPv6
node_expo 21651 nobody 321u sock 0,7 0t0 25192991 protocol: TCPv6
node_expo 21651 nobody 322u sock 0,7 0t0 25197884 protocol: TCPv6
node_expo 21651 nobody 323u sock 0,7 0t0 25195888 protocol: TCPv6
node_expo 21651 nobody 324u sock 0,7 0t0 25197887 protocol: TCPv6
node_expo 21651 nobody 325u sock 0,7 0t0 25192995 protocol: TCPv6
node_expo 21651 nobody 326u sock 0,7 0t0 25192020 protocol: TCPv6
node_expo 21651 nobody 327u sock 0,7 0t0 25192998 protocol: TCPv6
node_expo 21651 nobody 328u sock 0,7 0t0 25195895 protocol: TCPv6
node_expo 21651 nobody 329u sock 0,7 0t0 25195919 protocol: TCPv6
node_expo 21651 nobody 330u sock 0,7 0t0 25190995 protocol: TCPv6
node_expo 21651 nobody 331u sock 0,7 0t0 25193004 protocol: TCPv6
node_expo 21651 nobody 332u sock 0,7 0t0 25197906 protocol: TCPv6
node_expo 21651 nobody 333u sock 0,7 0t0 25197909 protocol: TCPv6
node_expo 21651 nobody 334u sock 0,7 0t0 25191000 protocol: TCPv6
node_expo 21651 nobody 335u IPv6 25191002 0t0 TCP nfjd.com:jetdirect->172.16.89.127:13481 (ESTABLISHED)
node_expo 21651 nobody 336u IPv6 25195961 0t0 TCP nfjd.com:jetdirect->172.16.89.192:19159 (ESTABLISHED)
node_expo 21651 nobody 337u IPv6 25193045 0t0 TCP nfjd.com:jetdirect->172.16.82.79:54286 (ESTABLISHED)
[root@nfjd log]#

[SuperQ]

Just to note, virtual memory is meaningless on 64-bit systems. The only thing that matters is RSS.

I would suggest adding nfs to the ignored filesystem list, and monitor NFS filesystems on the server side.

You can also try the new 0.17.0-rc.0 release which has a circuit breaker style patch for broken NFS filesystems.

[wfhu]

Just to note, virtual memory is meaningless on 64-bit systems. The only thing that matters is RSS.

Yes, virtual memory do not really consume memory resource. But the high virtual memory might be the reason of high system load. Maybe, I guess, because the huge memory space of the process? I think this is still a problem, which should be investigated thoroughly and solved.

I would suggest adding nfs to the ignored filesystem list, and monitor NFS filesystems on the server side.

You are correct, we are planning to ignore the nfs filesystem.

You can also try the new 0.17.0-rc.0 release which has a circuit breaker style patch for broken NFS filesystems.

We will try this new version once it is released officially. Hope the circuit breaker could prevent the same problem from happening.

[wfhu]

Seems to be related with #997

[SuperQ]

High virtual memory does not relate to high load average. Load average is not actually load, but a representation of the depth of the run queue. This is caused by the stuck OS threads waiting for NFS IO to un-deadlock. High load average only has a loose correlation with CPU utilization and "load" on server. It's a simplistic metric that is not really useful.

[SuperQ]

Duplicate of #868.