v0.14.0-rc2 breaks previously-working textfile metrics

[matthiasr]

We had metrics like

# TYPE smartmon_device_info gauge
smartmon_device_info{disk="/dev/bus/0",type="sat+megaraid,0",model_family="Seagate Constellation.2 (SATA)",device_model="ST91000640NS",serial_number="<redacted>",firmware_version="AA07"} 1
…
smartmon_device_info{disk="/dev/sda",type="scsi",vendor="DELL",product="PERC H710",revision="3.13",lun_id="<redacted>"} 1
…

(both repeated for 4 disks total)

from a script that parses smartctl output. The duplication is caused by devices being represented multiple times via the RAID controller.

With 0.14.0-rc2, we now get this error on the metrics endpoint:

An error has occurred during metrics gathering:

4 error(s) occurred:
* collected metric smartmon_device_info label:<name:"disk" value:"/dev/sda" > label:<name:"lun_id" value:"<redacted>" > label:<name:"product" value:"PERC H710" > label:<name:"revision" value:"3.13" > label:<name:"type" value:"scsi" > label:<name:"vendor" value:"DELL" > gauge:<value:1 >  has label dimensions inconsistent with previously collected metrics in the same metric family
…

I suspect this is a side effect of the vendoring update #372 bringing in stricter checks from client_golang.

The check and error are correct, we should not produce inconsistent metrics – however this does break metrics collection completely for previously working textfile metrics. This check should either be disabled or at least the breaking change clearly marked in the change log.

[beorn7]

As per personal discussion with @matthiasr :

The updated version of client_golang vendored with this node_exporter release does indeed improve consistency checking (as documented in the release notes of client_golang). The library always aimed for never creating inconsistent scrapes. It's one step closer to that goal now. It also allows more flexible configuration of how to deal with errors, see https://godoc.org/github.com/prometheus/client_golang/prometheus/promhttp#HandlerErrorHandling . So you could set ContinueOnError to expose as many metrics as possible. That might be the best option for the node_exporter. However, the library will still prevent inconsistent scrapes by not exposing the inconsistent metrics, i.e. first occurrence wins, later offenders are excluded from the scrape.