Node Exporter killed by SIGPIPE

[cristi-]

Host operating system: output of uname -a

Linux hostname 4.1.12-61.1.19.el7uek.x86_64 #2 SMP Fri Nov 18 17:00:09 PST 2016 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter -version

node_exporter, version 0.13.0 (branch: master, revision: 006d1c7)
build user: root@75db7098576a
build date: 20161126-13:11:09
go version: go1.7.3

Are you running node_exporter in Docker?

No

What did you do that produced an error?

Sometimes node_exporter dies on RHEL7 and on systemctl status we see this:

[root@hostname ~]# systemctl status node_exporter
● node_exporter.service - Prometheus exporter for machine metrics, written in Go with pluggable metric collectors.
Loaded: loaded (/usr/lib/systemd/system/node_exporter.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Sat 2017-05-27 04:54:39 EDT; 1 day 20h ago
Docs: https://prometheus.io
Main PID: 27203 (code=killed, signal=PIPE)

It seems to be somewhat related to this issue: moby/moby#7087 where when journald dies all go binaries to log to journald also die with SIGPIPE.

[root@hostname ~]# rpm -qa |grep systemd
systemd-219-30.0.1.el7_3.7.x86_64

[SuperQ]

I think this is correct behavior, as the stdout logging output pipe fails when journald is restarted. It's safe to add an automatic restart to your node exporter unit file.

[cristi-]

We have the folllwing systemd file:

/usr/lib/systemd/system/node_exporter.service

[Unit]
Description=Prometheus exporter for machine metrics, written in Go with pluggable metric collectors.
Documentation=https://prometheus.io
After=network.target

[Service]
EnvironmentFile=-/etc/sysconfig/node_exporter
User=prometheus
ExecStart=/usr/bin/node_exporter $ARGS
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure

But i think a PIPE signal is not considered an error.

What we are doing for other go programs is using https://github.com/projectatomic/forward-journald as a pipe between the program and journald.

Thanks!

[SuperQ]

I think that is unrelated, as that is for Go <= 1.6, where the version you are using is Go 1.7.

I don't think we've done anything with SIGPIPE in any other Prometheus code.

You probably want Restart=always.

[cristifalcas]

Actually it's a problem with go also. Some pertinent description on other project: cybozu-go/well#13.

Also: influxdata/kapacitor#737, moby/moby#19728.

In 1.6 they added support for SIGPIPE to be captured inside the program: https://golang.org/doc/go1.6

[SuperQ]

Thanks for the links, I guess we can look into handling SIGPIPE better.

[ymmt2005]

Hi,

I am the author of cybozu-go/cmd and would like to share my thoughts on this.

From a conclusion, the best way I think is to specify either Restart=always or
RestartForceExitStatus=SIGPIPE in the systemd service file.

We have struggled and failed twice to properly handle SIGPIPEs from journald
in Go, and finally resolved the problem in cybozu-go/well#17 . We chose to
address the problem within Go because cybozu-go/cmd and cybozu-go/log are
frameworks. By fixing the problem in the frameworks, any Go products built
on them will handle SIGPIPE nicely.

That said, our resolution is quite complex and not applicable for all cases.
If you can control systemd service definitions, the simplest way is to add either
Restart=always or RestartForceExitStatus=SIGPIPE in it.

Hope this helps.

[matthiasr]

So this is caused by the supervisor, and can easily be handled by the supervisor? In that case I would not add additional complexity to the exporter itself. This issue is a great reference, but I think we can close it. @SuperQ

[SuperQ]

Yes, thanks for all the info, but for now this is out of scope for anything the node_exporter needs to implement.