node_filesystem_device_error in v0.15.0 yet v0.14.0 works

[tmegow]

Host operating system: output of uname -a

Linux mongodb-backups 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 0.15.0 (branch: HEAD, revision: 6e2053c557f96efb63aef3691f15335a70baaffd)
  build user:       root@168089f37ad9
  build date:       20171006-11:33:58
  go version:       go1.9.1

node_exporter command line flags

--path.procfs /host/proc
--path.sysfs /host/sys
--collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/)"

Are you running node_exporter in Docker?

yes, with the recommended args

docker run
     --log-driver none
     --name node-exporter
     --net host
     --pid host
     --volume /proc:/host/proc
     --volume /sys:/host/sys
     --volume /:/rootfs
     --volume /etc/node-exporter:/etc/node-exporter
     prom/node-exporter
         --path.procfs /host/proc
         --path.sysfs /host/sys
         --collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/)"

What did you do that produced an error?

Trying to access exported disk metrics for k8s attached/mounted GCE persistent disk shows a node-exporter error metric incremented when node-exporter experienced an error while collecting statistics for this device.

# curl -s localhost:9100/metrics | grep mongodb-backups
node_filesystem_device_error{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/mongodb-backups"} 1
node_filesystem_device_error{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/pods/d0d4dfe6-b7fa-11e7-9da6-42010af0000a/volumes/kubernetes.io~gce-pd/mongodb-backups"} 1

What did you expect to see?

When using v0.14.0, I can see the filesystem metrics for that device/mountpoint (seen below). Yet the new version v0.15.0 does not export the same metrics.

docker run
     --log-driver none
     --name node-exporter
     --net host
     --pid host
     --volume /proc:/host/proc
     --volume /sys:/host/sys
     --volume /:/rootfs
     --volume /etc/node-exporter:/etc/node-exporter prom/node-exporter:v0.14.0
         -collector.procfs /host/proc
         -collector.sysfs /host/sys
         -collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/)"

# curl -s localhost:9100/metrics | grep mongodb-backups
node_filesystem_avail{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/mongodb-backups"} 9.3990998016e+10
node_filesystem_avail{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/pods/d0d4dfe6-b7fa-11e7-9da6-42010af0000a/volumes/kubernetes.io~gce-pd/mongodb-backups"} 9.3990998016e+10
node_filesystem_files{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/mongodb-backups"} 1.048576e+08
node_filesystem_files{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/pods/d0d4dfe6-b7fa-11e7-9da6-42010af0000a/volumes/kubernetes.io~gce-pd/mongodb-backups"} 1.048576e+08
node_filesystem_files_free{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/mongodb-backups"} 1.04857058e+08
node_filesystem_files_free{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/pods/d0d4dfe6-b7fa-11e7-9da6-42010af0000a/volumes/kubernetes.io~gce-pd/mongodb-backups"} 1.04857058e+08
node_filesystem_free{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/mongodb-backups"} 9.3990998016e+10
node_filesystem_free{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/pods/d0d4dfe6-b7fa-11e7-9da6-42010af0000a/volumes/kubernetes.io~gce-pd/mongodb-backups"} 9.3990998016e+10
node_filesystem_readonly{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/mongodb-backups"} 0
node_filesystem_readonly{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/pods/d0d4dfe6-b7fa-11e7-9da6-42010af0000a/volumes/kubernetes.io~gce-pd/mongodb-backups"} 0
node_filesystem_size{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/mongodb-backups"} 1.073217536e+11
node_filesystem_size{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/pods/d0d4dfe6-b7fa-11e7-9da6-42010af0000a/volumes/kubernetes.io~gce-pd/mongodb-backups"} 1.073217536e+11

What did you see instead?

A metric implying that node-exporter encountered an error while collecting statistics for this device.

# curl -s localhost:9100/metrics | grep mongodb-backups
node_filesystem_device_error{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/mongodb-backups"} 1
node_filesystem_device_error{device="/dev/sdb",fstype="xfs",mountpoint="/rootfs/var/lib/kubelet/pods/d0d4dfe6-b7fa-11e7-9da6-42010af0000a/volumes/kubernetes.io~gce-pd/mongodb-backups"} 1

[tmegow]

One more thing to note, with v0.15.0 this symptom is happening with all GCE PD mounts in our kube clusters (even on our Google Container-Optimized OS instances).

Linux minion-47pw 4.4.21+ #1 SMP Fri Feb 17 15:34:45 PST 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux

# cat /etc/*release
CHROMEOS_AUSERVER=https://tools.google.com/service/update2
CHROMEOS_BOARD_APPID={76E245CF-C0D0-444D-BA50-36739C18EB00}
CHROMEOS_CANARY_APPID={90F229CE-83E2-4FAF-8479-E368A34938B1}
CHROMEOS_DEVSERVER=
CHROMEOS_RELEASE_APPID={76E245CF-C0D0-444D-BA50-36739C18EB00}
CHROMEOS_RELEASE_BOARD=lakitu-signed-mpkeys
CHROMEOS_RELEASE_BRANCH_NUMBER=84
CHROMEOS_RELEASE_BUILDER_PATH=lakitu-release/R56-9000.84.2
CHROMEOS_RELEASE_BUILD_NUMBER=9000
CHROMEOS_RELEASE_BUILD_TYPE=Official Build
CHROMEOS_RELEASE_CHROME_MILESTONE=56
CHROMEOS_RELEASE_DESCRIPTION=9000.84.2 (Official Build) stable-channel lakitu
CHROMEOS_RELEASE_NAME=Chrome OS
CHROMEOS_RELEASE_PATCH_NUMBER=2
CHROMEOS_RELEASE_TRACK=stable-channel
CHROMEOS_RELEASE_VERSION=9000.84.2
DEVICETYPE=OTHER
GOOGLE_RELEASE=9000.84.2
HWID_OVERRIDE=LAKITU DEFAULT
BUILD_ID=9000.84.2
NAME="Container-Optimized OS"
GOOGLE_CRASH_ID=Lakitu
VERSION_ID=56
BUG_REPORT_URL=https://crbug.com/new
PRETTY_NAME="Container-Optimized OS from Google"
VERSION=56
GOOGLE_METRICS_PRODUCT_ID=26
HOME_URL="https://cloud.google.com/compute/docs/containers/vm-image/"
ID=cos

[discordianfish]

Thanks for the detailed report! Would be most interested in the log output of the node-exporter. Can you provide that too?

[tmegow]

Sure, here are the logs for v0.15.0

time="2017-10-25T21:41:47Z" level=info msg="Starting node_exporter (version=0.15.0, branch=HEAD, revision=6e2053c557f96efb63aef3691f15335a70baaffd)" source="node_exporter.go:43"
time="2017-10-25T21:41:47Z" level=info msg="Build context (go=go1.9.1, user=root@168089f37ad9, date=20171006-11:33:58)" source="node_exporter.go:44"
time="2017-10-25T21:41:47Z" level=info msg="Enabled collectors:" source="node_exporter.go:50"
time="2017-10-25T21:41:47Z" level=info msg=" - time" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - conntrack" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - infiniband" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - entropy" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - cpu" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - edac" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - wifi" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - vmstat" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - netstat" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - loadavg" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - bcache" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - sockstat" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - zfs" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - timex" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - mdadm" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - meminfo" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - netdev" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - uname" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - filefd" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - filesystem" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - xfs" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - arp" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - hwmon" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - ipvs" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - stat" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - textfile" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg=" - diskstats" source="node_exporter.go:52"
time="2017-10-25T21:41:47Z" level=info msg="Listening on :9100" source="node_exporter.go:76"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/5c1a8db44efd276057e1ec62d10e130aa1f9e1685fca1d9ecde0821f09d117ec/merged\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/proc\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/dev\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/dev/pts\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/dev/mqueue\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/systemd\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/pids\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/perf_event\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/freezer\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/cpuset\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/net_cls,net_prio\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="ERROR: timex collector failed after 0.000441s: failed to retrieve adjtimex stats: operation not permitted" source="collector.go:123"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/cpu,cpuacct\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/memory\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/blkio\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/devices\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/overlay/bd937f278b2c580c2897139e771f57aae24215476da4760b4e0aa0f7494f5961/merged/sys/fs/cgroup/hugetlb\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/dev\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/dev/shm\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/dev/pts\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/dev/mqueue\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/dev/hugepages\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/proc\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/kernel/security\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/systemd\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/pids\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/perf_event\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/freezer\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/cpuset\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/net_cls,net_prio\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/cpu,cpuacct\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/memory\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/blkio\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/devices\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/cgroup/hugetlb\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/pstore\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/fs/selinux\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/sys/kernel/debug\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/tmp\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/etc\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/home\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/rkt/pods/run/78c92685-c86b-43b5-a30a-71ad37fa7ef9/stage1/rootfs/opt/stage2/oem-gce/rootfs/run/systemd\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/containers/3f40c3188fa9c87ae7c5f5ac3031954e59f1213a098b1d592a0a0e41cc33b769/shm\": permission denied" source="filesystem_linux.go:57"
time="2017-10-25T21:41:47Z" level=error msg="Error on statfs() system call for \"/rootfs/var/lib/docker/containers/ac1e9ee6756fe150674431125ebea644650f3fcea0370f3bf830ebfbebdd7c0b/shm\": permission denied" source="filesystem_linux.go:57"

In contrast, v0.14.0 has fewer permission denied errors:

time="2017-10-25T21:45:12Z" level=info msg="Starting node_exporter (version=0.14.0, branch=master, revision=840ba5dcc71a084a3bc63cb6063003c1f94435a6)" source="node_exporter.go:140"
time="2017-10-25T21:45:12Z" level=info msg="Build context (go=go1.7.5, user=root@bb6d0678e7f3, date=20170321-12:12:54)" source="node_exporter.go:141"
time="2017-10-25T21:45:12Z" level=info msg="Enabled collectors:" source="node_exporter.go:160"
time="2017-10-25T21:45:12Z" level=info msg=" - diskstats" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - infiniband" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - loadavg" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - stat" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - wifi" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - filesystem" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - netdev" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - textfile" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - uname" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - vmstat" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - entropy" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - edac" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - filefd" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - sockstat" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - zfs" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - time" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - conntrack" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - hwmon" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - mdadm" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - meminfo" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg=" - netstat" source="node_exporter.go:162"
time="2017-10-25T21:45:12Z" level=info msg="Listening on :9100" source="node_exporter.go:186"

[discordianfish]

Thanks! This looks like the same as #703, lets discuss over there.

[discordianfish]

Okay so the root cause is the same: We're dropping privileges to nobody now, which means you can't statsfs() the bind-mounted mountpoints. If you want to monitor the mongodb-backups volume you should mount it to the node-exporter pod. Then you also need to ignore all of these failing mountpoints, e.g:

      args:
          - --collector.filesystem.ignored-mount-points
          - '^\/rootfs\/(var\/lib|run\/docker)\/.*'

If you don't want to mount the volume to the node-exporter pod, you can of course also force it to run as root like described here: #599 (comment)