node_exporter with Textfile directory metrics collection hangs #743
Description
Host operating system: output of uname -a
Linux hostname 3.10.0-514.16.1.el7.x86_64 #1 SMP Fri Mar 10 13:12:32 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
node_exporter version: output of node_exporter --version
/tmp/node_exporter --version
node_exporter, version 0.15.0 (branch: master, revision: a8d7d11)
build user: root@hostname
build date: 20171118-20:08:56
go version: go1.9.2
node_exporter command line flags
/tmp/node_exporter --collector.textfile.directory="/var/tmp/thin" &
Are you running node_exporter in Docker?
NO, running locally on the base host, not in a container
What did you do that produced an error?
let the collector run for over an hour and the process hangs...
What did you expect to see?
collector should not hang...
What did you see instead?
collector hangs, consistently across multiple systems.
Summary
/tmp/node_exporter --collector.textfile.directory="/var/tmp/thin" &
"/var/tmp/thin" folders contains *.prom files for parsing...
cat /var/tmp/thin/lvm.prom
node_lvm_thinpool_size{lvname="thinpool",vgname="docker"} 64.91
node_lvm_thinpool_used{lvname="thinpool",vgname="docker"} 19.52
after initial startup, exporter works fine, prometheus scrapes this end point sucessfull including the metrics from the *.prom file.
after about an hour or so, the end point becomes unresponsive.....
node_exporter PID on the host is still up but unresponsive..
root 22284 8947 20 12:19 pts/0 00:44:49 /tmp/node_exporter --collector.textfile.directory=/var/tmp/thin
tracing the collector pid does not return anything...
# strace -p 22284
Process 22284 attached
futex(0x1064530, FUTEX_WAIT, 0, NULL
^CProcess 22284 detached
<detached ...>
the session on which the node_exporter process is dumps barrage of messages as shown as below....
<..snip..>
Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1]
[from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1]
[from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1]
[from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1]
[from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1]
[from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1]
[from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1]
[from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1]
[from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1]
[from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1] [from Gatherer #1]
[from Gatherer #1] [from Gatherer #1] colle[1]+ Killed /tmp/node_exporter --collector.textfile.directory="/var/tmp/thin"
eventually the process needs to be killed
# kill -9 22284