hwmon duplicate core temps

[mgulick]

On node_exporter 13.0-rc.1 on Linux (Debian 8), I'm getting the following warnings from prometheus:

Oct 26 09:40:08 sbtools-11-ah prometheus[9625]: time="2016-10-26T09:40:08-04:00" level=debug msg="Sample discarded" error="sample with repeated timestamp but different value" sample=node_hwmon_temp_celsius{chip="coretemp", instance="sb6934glnxa64:9100", job="sbfarm-nodes", sensor="core_0"} => 54 @[1477489208.369] source="scrape.go:460"
Oct 26 09:40:08 sbtools-11-ah prometheus[9625]: time="2016-10-26T09:40:08-04:00" level=debug msg="Sample discarded" error="sample with repeated timestamp but different value" sample=node_hwmon_temp_celsius{chip="coretemp", instance="sb6934glnxa64:9100", job="sbfarm-nodes", sensor="core_1"} => 72 @[1477489208.369] source="scrape.go:460"
Oct 26 09:40:08 sbtools-11-ah prometheus[9625]: time="2016-10-26T09:40:08-04:00" level=debug msg="Sample discarded" error="sample with repeated timestamp but different value" sample=node_hwmon_temp_celsius{chip="coretemp", instance="sb6934glnxa64:9100", job="sbfarm-nodes", sensor="core_10"} => 74 @[1477489208.369] source="scrape.go:460"
Oct 26 09:40:08 sbtools-11-ah prometheus[9625]: time="2016-10-26T09:40:08-04:00" level=debug msg="Sample discarded" error="sample with repeated timestamp but different value" sample=node_hwmon_temp_celsius{chip="coretemp", instance="sb6934glnxa64:9100", job="sbfarm-nodes", sensor="core_11"} => 68 @[1477489208.369] source="scrape.go:460"
Oct 26 09:40:08 sbtools-11-ah prometheus[9625]: time="2016-10-26T09:40:08-04:00" level=debug msg="Sample discarded" error="sample with repeated timestamp but different value" sample=node_hwmon_temp_celsius{chip="coretemp", instance="sb6934glnxa64:9100", job="sbfarm-nodes", sensor="core_12"} => 72 @[1477489208.369] source="scrape.go:460"
Oct 26 09:40:08 sbtools-11-ah prometheus[9625]: time="2016-10-26T09:40:08-04:00" level=debug msg="Sample discarded" error="sample with repeated timestamp but different value" sample=node_hwmon_temp_celsius{chip="coretemp", instance="sb6934glnxa64:9100", job="sbfarm-nodes", sensor="core_2"} => 74 @[1477489208.369] source="scrape.go:460"
Oct 26 09:40:08 sbtools-11-ah prometheus[9625]: time="2016-10-26T09:40:08-04:00" level=debug msg="Sample discarded" error="sample with repeated timestamp but different value" sample=node_hwmon_temp_celsius{chip="coretemp", instance="sb6934glnxa64:9100", job="sbfarm-nodes", sensor="core_3"} => 56 @[1477489208.369] source="scrape.go:460"
Oct 26 09:40:08 sbtools-11-ah prometheus[9625]: time="2016-10-26T09:40:08-04:00" level=debug msg="Sample discarded" error="sample with repeated timestamp but different value" sample=node_hwmon_temp_celsius{chip="coretemp", instance="sb6934glnxa64:9100", job="sbfarm-nodes", sensor="core_4"} => 76 @[1477489208.369] source="scrape.go:460"
Oct 26 09:40:08 sbtools-11-ah prometheus[9625]: time="2016-10-26T09:40:08-04:00" level=debug msg="Sample discarded" error="sample with repeated timestamp but different value" sample=node_hwmon_temp_celsius{chip="coretemp", instance="sb6934glnxa64:9100", job="sbfarm-nodes", sensor="core_8"} => 65 @[1477489208.369] source="scrape.go:460"
Oct 26 09:40:08 sbtools-11-ah prometheus[9625]: time="2016-10-26T09:40:08-04:00" level=debug msg="Sample discarded" error="sample with repeated timestamp but different value" sample=node_hwmon_temp_celsius{chip="coretemp", instance="sb6934glnxa64:9100", job="sbfarm-nodes", sensor="core_9"} => 73 @[1477489208.369] source="scrape.go:460"

A sample from /metrics:

# HELP node_hwmon_temp_celsius Hardware monitor for temperature (input)
# TYPE node_hwmon_temp_celsius gauge
node_hwmon_temp_celsius{chip="coretemp",sensor="core_0"} 65
node_hwmon_temp_celsius{chip="coretemp",sensor="core_0"} 62
node_hwmon_temp_celsius{chip="coretemp",sensor="core_1"} 62
node_hwmon_temp_celsius{chip="coretemp",sensor="core_1"} 63
node_hwmon_temp_celsius{chip="coretemp",sensor="core_10"} 62
node_hwmon_temp_celsius{chip="coretemp",sensor="core_10"} 62
node_hwmon_temp_celsius{chip="coretemp",sensor="core_11"} 66
node_hwmon_temp_celsius{chip="coretemp",sensor="core_11"} 61
...

CPU is a Xeon E5-2680 v3

Let me know if there is any other information I can provide.

[brian-brazil]

@rtreffer

[SuperQ]

It looks like this is an issue for dual socket hardware, there are two coretemp trees on these nodes.

[SuperQ]

I think we need to include the device path as a label.

For example:

/sys/class/hwmon$ ls -l hwmon*/device
lrwxrwxrwx 1 root root 0 Oct 26 15:04 hwmon0/device -> ../../../0000:02:00.0
lrwxrwxrwx 1 root root 0 Oct 26 15:05 hwmon1/device -> ../../../0000:06:00.0
lrwxrwxrwx 1 root root 0 Oct 26 15:00 hwmon2/device -> ../../../coretemp.0
lrwxrwxrwx 1 root root 0 Oct 26 15:01 hwmon3/device -> ../../../coretemp.1

[rtreffer]

Ah, I suspected this might happen. @brian-brazil I removed the path-derived name which is guaranteed to be unique, should I re-add that to the labels?

[brian-brazil]

So that'd be coretemp.1 here rather than coretemp?

[rtreffer]

Both solutions can lead to problems.....

Using path fragments

Pro: Guaranteed to be unique
Con: Can change between kernel version or even reboots. Possibly cryptic (e.g. isa-0000:02:00.0)

Using names as exported

Pro: Human readable, stable across reboots
Con: Might be not-unique

(I actually tried to come up with a situation where it is non-unique, but failed to come up with this one shown here)

My feeling right now would be to use the device path/name if a device link exists, then switching to the exported name and finally using the hwmon name.
Basically just changing the precedence in https://github.com/prometheus/node_exporter/blob/master/collector/hwmon_linux.go#L296

[rtreffer]

And that would use platform-coretemp.0 as the name. Which is always unique.

[brian-brazil]

Sounds like a plan.