Couldn't get system state: Unknown property

[drewhemm]

This is appearing frequently in /var/log/messages:

Aug 25 12:46:18 bamboo node_exporter: time="2016-08-25T12:46:18Z" level=error msg="ERROR: systemd collector failed after 0.009243s: couldn't get system state: Unknown property" source="node_exporter.go:91"

Not sure if I should care and/or how to silence it.

[drewhemm]

After some initial investigation, it appears that the machines that are not complaining are running a higher version of systemd - 219 vs 208. The higher version outputs systemctl status differently, and the output includes a property 'State'. The question is, is the difference in output solely caused the difference in version (and if so, is the systemd collector only expected to work on machines with a certain version of systemd?) or is it caused by something else?

Working machine:

[ahemming@cms-####### ~]$ systemctl status
● cms-#######
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Wed 2016-08-24 10:36:59 UTC; 1 day 2h ago
...

Machine that complains:

[ahemming@bamboo ~]$ systemctl status
proc-sys-fs-binfmt_misc.automount -> '/org/freedesktop/systemd1/unit/proc_2dsys_2dfs_2dbinfmt_5fmisc_2eautomount'
proc-sys-fs-binfmt_misc.automount - Arbitrary Executable File Formats File System Automount Point
   Loaded: loaded (/usr/lib/systemd/system/proc-sys-fs-binfmt_misc.automount; static)
   Active: active (running) since Thu 2016-04-07 16:13:58 UTC; 4 months 18 days ago
    Where: /proc/sys/fs/binfmt_misc
     Docs: https://www.kernel.org/doc/Documentation/binfmt_misc.txt
           http://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
...

[discordianfish]

Which version of the node-exporter are you using?
The collector is not parsing systemctl output but using go-systemd/dbus. The error is returned by conn.GetManagerProperty("SystemState") after successfully connecting to dbus. Maybe that propery doesn't exist in 208?

Can you run this and see what it's reporting?

package main

import (
	"log"

	"github.com/coreos/go-systemd/dbus"
)

func main() {
	conn, err := dbus.New()
	if err != nil {
		log.Fatal(err)
	}
	state, err := conn.GetManagerProperty("SystemState")
	if err != nil {
		log.Fatal(err)
	}
	log.Printf("state: %v", state)
}

If you can't compile it yourself, here is a linux/amd64 binary: https://drive.google.com/file/d/0B5Dqy-L1tbQjeU1jeVZXX3NlT3c/view?usp=sharing

@fabxc You probably have good systemd connections: Any ideas why this might be missing? Maybe it was introduced after 208 but haven't found anything in the changelog.

[pengsrc]

@discordianfish Hi, I got Unknown property when executing this code, this should be a bug in go-systemd.

Here's the example of my crond service status.

# systemctl status crond.service
crond.service - Command Scheduler
   Loaded: loaded (/usr/lib/systemd/system/crond.service; enabled)
   Active: active (running) since Mon 2016-08-29 16:14:16 CST; 8 months 1 days ago
 Main PID: 434 (crond)
   CGroup: /system.slice/crond.service
           └─434 /usr/sbin/crond -n

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

May be the go-systemd package can not parse a service if it has been running for a long time and it's log is not available.

[SuperQ]

Is this still an issue with the latest release?

[beylisstan]

Same issue here.

Host operating system: output of uname -a

Linux xx-xxx-xxxxxx-0 3.12.62-60.62-default #1 SMP Thu Aug 4 09:06:08 UTC 2016 (b0e5a26) x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 0.15.2 (branch: HEAD, revision: 98bc649)
build user: root@d5c4792c921f
build date: 20171205-14:50:53
go version: go1.9.2

node_exporter command line flags

./node_exporter --collector.systemd

Are you running node_exporter in Docker?

No

What did you do that produced an error?

./node_exporter --collector.systemd &

What did you expect to see?

Running exporter without errors

What did you see instead?

INFO[0000] Starting node_exporter (version=0.15.2, branch=HEAD, revision=98bc64930d34878b84a0f87dfe6e1a6da61e532d) source="node_exporter.go:43"
INFO[0000] Build context (go=go1.9.2, user=root@d5c4792c921f, date=20171205-14:50:53) source="node_exporter.go:44"
INFO[0000] No directory specified, see --collector.textfile.directory source="textfile.go:57"
INFO[0000] Enabled collectors: source="node_exporter.go:50"
INFO[0000] - stat source="node_exporter.go:52"
INFO[0000] - entropy source="node_exporter.go:52"
INFO[0000] - cpu source="node_exporter.go:52"
INFO[0000] - loadavg source="node_exporter.go:52"
INFO[0000] - netstat source="node_exporter.go:52"
INFO[0000] - uname source="node_exporter.go:52"
INFO[0000] - arp source="node_exporter.go:52"
INFO[0000] - conntrack source="node_exporter.go:52"
INFO[0000] - zfs source="node_exporter.go:52"
INFO[0000] - filefd source="node_exporter.go:52"
INFO[0000] - edac source="node_exporter.go:52"
INFO[0000] - filesystem source="node_exporter.go:52"
INFO[0000] - infiniband source="node_exporter.go:52"
INFO[0000] - diskstats source="node_exporter.go:52"
INFO[0000] - meminfo source="node_exporter.go:52"
INFO[0000] - bcache source="node_exporter.go:52"
INFO[0000] - hwmon source="node_exporter.go:52"
INFO[0000] - sockstat source="node_exporter.go:52"
INFO[0000] - netdev source="node_exporter.go:52"
INFO[0000] - mdadm source="node_exporter.go:52"
INFO[0000] - time source="node_exporter.go:52"
INFO[0000] - vmstat source="node_exporter.go:52"
INFO[0000] - ipvs source="node_exporter.go:52"
INFO[0000] - wifi source="node_exporter.go:52"
INFO[0000] - systemd source="node_exporter.go:52"
INFO[0000] - timex source="node_exporter.go:52"
INFO[0000] - textfile source="node_exporter.go:52"
INFO[0000] - xfs source="node_exporter.go:52"
INFO[0000] Listening on :9100 source="node_exporter.go:76"
ERRO[0003] ERROR: systemd collector failed after 0.018945s: couldn't get system state: Unknown property or interface. source="collector.go:123"
ERRO[0004] ERROR: systemd collector failed after 0.020527s: couldn't get system state: Unknown property or interface. source="collector.go:123"

[discordianfish]

@beylisstan Can you try if this still happens with 0.16.0?

[discordianfish]

I believe this has been fixed in the meanwhile. If not, please re-open this issue.

[tiagoapimenta]

Could you please reopen this bug?

Environment:

uname -a
Linux xxxx 3.10.0-229.el7.x86_64 #1 SMP Thu Jan 29 18:37:38 EST 2015 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/system-release
Red Hat Enterprise Linux Server release 7.1 (Maipo)

node-exporter --version
node_exporter, version 0.18.1 (branch: HEAD, revision: 3db77732e925c08f675d7404a8c46466b2ece83e)
  build user:       root@b50852a1acba
  build date:       20190604-16:41:18
  go version:       go1.12.5

Here is some evidence:

sudo node-exporter --log.level=debug --web.listen-address=":9101" --collector.systemd --collector.textfile.directory=/var/lib/node-porter
...
DEBU[0004] Ignoring unit: dev-disk-by\x2duuid-f6aafd9c\x2d1d30\x2d4d12\x2d9e95\x2d66a23870ac09.device  source="systemd_linux.go:421"
DEBU[0004] Ignoring unit: dev-mapper-vg_app\x2dlv_gemsys.device  source="systemd_linux.go:421"
DEBU[0004] Adding unit: polkit.service                   source="systemd_linux.go:418"
DEBU[0004] Ignoring unit: user-1000.slice                source="systemd_linux.go:421"
DEBU[0004] Ignoring unit: user-0.slice                   source="systemd_linux.go:421"
DEBU[0004] Ignoring unit: session-35.scope               source="systemd_linux.go:421"
DEBU[0004] systemd filterUnits took 0.006092             source="systemd_linux.go:149"
DEBU[0004] systemd collectSystemState took 0.001317      source="systemd_linux.go:200"
DEBU[0004] couldn't get unit 'systemd-readahead-done.timer' LastTriggerUSec: Unknown property  source="systemd_linux.go:342"
DEBU[0004] couldn't get unit 'systemd-tmpfiles-clean.timer' LastTriggerUSec: Unknown property  source="systemd_linux.go:342"
DEBU[0004] systemd collectTimers took 0.004424           source="systemd_linux.go:187"
DEBU[0004] systemd collectSockets took 0.035076          source="systemd_linux.go:195"
DEBU[0004] systemd collectUnitStatusMetrics took 0.088215  source="systemd_linux.go:159"
ERRO[0004] ERROR: systemd collector failed after 0.110036s: couldn't get system state: Unknown property  source="collector.go:132"

sudo systemctl status systemd-readahead-done.timer systemd-tmpfiles-clean.timer
systemd-readahead-done.timer - Stop Read-Ahead Data Collection 10s After Completed Startup
   Loaded: loaded (/usr/lib/systemd/system/systemd-readahead-done.timer; static)
   Active: inactive (dead)
           start condition failed at Sex 2019-06-07 15:00:34 -03; 3h 8min ago
           ConditionVirtualization=no was not met
     Docs: man:systemd-readahead-replay.service(8)

systemd-tmpfiles-clean.timer - Daily Cleanup of Temporary Directories
   Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-clean.timer; static)
   Active: active (waiting) since Sex 2019-06-07 14:58:47 -03; 3h 10min ago
     Docs: man:tmpfiles.d(5)
           man:systemd-tmpfiles(8)

[SuperQ]

It looks like this is coming from collectSystemState().

cc: @pgier