ERROR: cpufreq collector failed after 1.499322s: strconv.ParseUint: parsing \"<unknown>\": invalid syntax" source="collector.go:132" #1710
Comments
|
This seems like a duplicate of #1241, but possibly a new bug. Can you run the node_exporter with --log.level=debug and get the output for the cpufreq collector? |
|
It might also be worth trying out the latest 1.0 release candidate. |
|
|
|
This looks like an issue with the procfs parsing of the cpufreq data. |
|
It would be useful to include the text output of this script: |
|
for f in $(find /sys/devices/system/cpu/cpu*/cpufreq/ -type f) ; do echo "$f: $(< $f)" ; done |
|
That's strange, I don't see |
|
yes, I'm sure，the bug is a rare occurrence |
|
I take it that means the error doesn't occur every scrape? How often does this error get reported? |
|
We have been continuously seeing this as well (still node_exporter 0.18.x). Across a fleet of about 500 HPE DL380 Gen8/9/10 servers running on RHEL7, there are almost always some (5-10) servers in a state where the cpufreq collector returns a failure with the log showing exactly this error message. My gut feeling says that some machines are more affected than others (maybe CPU type / core count?), but I have not made any analyses to verify this (yet). I suspect a kernel and/or firmware/microcode bug. If I remember correctly, this problem started to increase over time which would imply that some kind of update broke it. We run latest RHEL7 while the original report is about a rather old RHEL7/CentOS7 kernel (7.0/1/2?). Our workaround consists of accepting the gaps + setting a longer node_exporter/Prometheus makes this problem visible (and annoying), but I'm not sure if there's anything fixable on the node_exporter/procfs side if the kernel doesn't return valid data. Please take all of this with a grain of salt as I have no hard evidence to back this. However, I'm not sure if we ever get such evidence as we don't have a way to reproduce this reliably at will. |
|
Thanks @hoffie. If this was a consistent problem where it always returned It's an indicator of a real problem. Covering it up is more dangerous than the minor annoyance of some missed data and soft errors in the scrape. We already handle the collector failing in a soft way. I do think the error message could be improved, to make it easier to see exactly what CPU/metric failed. |
|
On the affected systems, the following loop demonstrates the problem outside of node_exporter after some time: We have opened a support case with Red Hat and at least on our HPE machines running RHEL7 this issue can be explained by a broken cpufreq driver. We have confirmed that changing the BIOS setting makes Linux load the intel_pstate driver. We have not yet run any larger tests if this fixes the problem described in this Github issue, but I'm optimistic. :) So, the generic answer is as expected: This is a problem in the cpufreq driver and nothing which node_exporter can fix. It also seems that this cosmetic issue is one of the less important symptoms of the buggy driver (which, it appears, can lead to unbootable or performance-degraded systems). References:
@SuperQ I had a quick look regarding improving the error messages. node_exporter uses procfs' sysfs package and its parseCpufreqCpuinfo function. That function uses ReadUintFromFile. So I assume this is where the error should be wrapped with a hint about the affected filename. If this is wanted, I assume that it should be done with all those helper functions which take a path name, right? |
|
Thanks for the detailed info. Yes, improving things in the procfs code would be the right place. |
This improves error messages by adding the affected file name. This helps downstream users such as node_exporter to locate the source of errors which is useful for prometheus/node_exporter#1710. The original error is wrapped as some consumers perform checks on it. The consumers within procfs have been updated to Unwrap the error if necessary. Signed-off-by: Christian Hoffmann <[email protected]>
This improves error messages by adding the affected file name. This helps downstream users to locate the source of errors which is useful for e.g. prometheus/node_exporter#1710. The original error is wrapped as some consumers perform checks on it. The consumers within procfs have been updated to Unwrap the error if necessary. Signed-off-by: Christian Hoffmann <[email protected]>
I'v met this issue with node_exporter 1.1.4. My enviroment is a centos7.6 vm guest. The output of command metioned before: |
|
Just a quick update from my side: As expected, this issue went away for us by fixing the frequency scaling driver as recommended by the vendors (i.e. moving away from pcc-cpufreq). So, it was not a node_exporter issue, but a kernel (driver) bug in our case. |
Throw if file has non-numeric data, in addition to hinting the path that caused the error. Fixes: prometheus#304, prometheus/node_exporter#1710 Signed-off-by: Pranshu Srivastava <[email protected]>
Throw if file has non-numeric data, in addition to hinting the path that caused the error. Fixes: prometheus#304, prometheus/node_exporter#1710 Signed-off-by: Pranshu Srivastava <[email protected]>
|
Raised a PR on |
Host operating system: output of
uname -aLinux EM-4V8NH42 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
node_exporter version: output of
node_exporter --versionnode_exporter, version 0.18.1 (branch: HEAD, revision: 3db7773)
build user: root@b50852a1acba
build date: 20190604-16:41:18
go version: go1.12.5
node_exporter command line flags
/opt/node_exporter-0.18.1.linux-amd64/node_exporter --web.listen-address=":9111"
Are you running node_exporter in Docker?
no
How to solve or avoid this problem? Thank you
The text was updated successfully, but these errors were encountered: