os-node_exporter collector failed for zfs with err="couldn't get sysctl: no such file or directory"

[berlinbikerx]

After updating my OPNsense installation to version 20.1.8_1-amd64 I constantly get the following error message in system.log:

Jul 21 00:36:59 opnsense node_exporter[88175]: level=error ts=2020-07-20T22:36:59.251Z caller=collector.go:161 msg="collector failed" name=zfs duration_seconds=1.4879e-05 err="couldn't get sysctl: no such file or directory" 

Software versions are as follows:
OPNsense 20.1.8_1-amd64
FreeBSD 11.2-RELEASE-p20-HBSD
OpenSSL 1.1.1g 21 Apr 2020

node_exporter version=1.0.1, branch=release-1.0, revision=0

The startup log of node_exporter is

Jul 21 00:36:55 opnsense node_exporter[88175]: level=info ts=2020-07-20T22:36:55.550Z caller=node_exporter.go:177 msg="Starting node_exporter" version="(version=1.0.1, branch=release-1.0, revision=0)" 
Jul 21 00:36:55 opnsense node_exporter[88175]: level=info ts=2020-07-20T22:36:55.550Z caller=node_exporter.go:178 msg="Build context" build_context="(go=go1.14.4, user=root, date=)" 
Jul 21 00:36:55 opnsense node_exporter[88175]: level=info ts=2020-07-20T22:36:55.550Z caller=node_exporter.go:105 msg="Enabled collectors" 
Jul 21 00:36:55 opnsense node_exporter[88175]: level=info ts=2020-07-20T22:36:55.551Z caller=node_exporter.go:112 collector=boottime 
Jul 21 00:36:55 opnsense node_exporter[88175]: level=info ts=2020-07-20T22:36:55.551Z caller=node_exporter.go:112 collector=cpu 
Jul 21 00:36:55 opnsense node_exporter[88175]: level=info ts=2020-07-20T22:36:55.551Z caller=node_exporter.go:112 collector=devstat 
Jul 21 00:36:55 opnsense node_exporter[88175]: level=info ts=2020-07-20T22:36:55.551Z caller=node_exporter.go:112 collector=exec 
Jul 21 00:36:55 opnsense node_exporter[88175]: level=info ts=2020-07-20T22:36:55.551Z caller=node_exporter.go:112 collector=filesystem 
Jul 21 00:36:55 opnsense node_exporter[88175]: level=info ts=2020-07-20T22:36:55.551Z caller=node_exporter.go:112 collector=loadavg 
Jul 21 00:36:55 opnsense node_exporter[88175]: level=info ts=2020-07-20T22:36:55.551Z caller=node_exporter.go:112 collector=meminfo 
Jul 21 00:36:55 opnsense node_exporter[88175]: level=info ts=2020-07-20T22:36:55.551Z caller=node_exporter.go:112 collector=netdev 
Jul 21 00:36:55 opnsense node_exporter[88175]: level=info ts=2020-07-20T22:36:55.551Z caller=node_exporter.go:112 collector=textfile 
Jul 21 00:36:55 opnsense node_exporter[88175]: level=info ts=2020-07-20T22:36:55.551Z caller=node_exporter.go:112 collector=time 
Jul 21 00:36:55 opnsense node_exporter[88175]: level=info ts=2020-07-20T22:36:55.551Z caller=node_exporter.go:112 collector=uname 
Jul 21 00:36:55 opnsense node_exporter[88175]: level=info ts=2020-07-20T22:36:55.551Z caller=node_exporter.go:112 collector=zfs 
Jul 21 00:36:55 opnsense node_exporter[88175]: level=info ts=2020-07-20T22:36:55.551Z caller=node_exporter.go:191 msg="Listening on" address=0.0.0.0:9100 
Jul 21 00:36:55 opnsense node_exporter[88175]: level=info ts=2020-07-20T22:36:55.551Z caller=tls_config.go:170 msg="TLS is disabled and it cannot be enabled on the fly." http2=false 
Jul 21 00:36:56 opnsense node_exporter[88175]: level=error ts=2020-07-20T22:36:56.268Z caller=collector.go:161 msg="collector failed" name=zfs duration_seconds=2.6973e-05 err="couldn't get sysctl: no such file or directory" 
Jul 21 00:36:57 opnsense node_exporter[88175]: level=error ts=2020-07-20T22:36:57.251Z caller=collector.go:161 msg="collector failed" name=zfs duration_seconds=1.555e-05 err="couldn't get sysctl: no such file or directory" 
Jul 21 00:36:58 opnsense node_exporter[88175]: level=error ts=2020-07-20T22:36:58.251Z caller=collector.go:161 msg="collector failed" name=zfs duration_seconds=1.7693e-05 err="couldn't get sysctl: no such file or directory" 

[berlinbikerx]

I just updated to the latest version OPNsense 20.1.9-amd64, the messages are still in the logs.

[fichtner]

Looks like an issue updating node_exporter from 0.18.1 (20.1.7) to 1.0.1 (20.1.8 and up). Your best bet is to poke the authors of the software...

[berlinbikerx]

I could mitigate the issue by adding the following line to the command line argumentes of /usr/local/etc/rc.d/node_exporter:

--no-collector.zfs

The release notes for node_exporter 1.0.0 state, that ZFS support was added for FreeBSD, (cf. prometheus/node_exporter#1714 and prometheus/node_exporter#1394)

As that was not available during 0.18.1, it could be an option to disable the ZFS collector per default in the OPNsense package or add a configuration option in the UI. What do you think?

[lrosenman]

I think this would be a good thing to add to the OPNSense package. Either disable by default or make an option to enable it but still default to off.
IMO, of course.

[OPNsense-bot]

This issue has been automatically timed-out (after 180 days of inactivity).

For more information about the policies for this repository,
please read https://github.com/opnsense/plugins/blob/master/CONTRIBUTING.md for further details.

If someone wants to step up and work on this issue,
just let us know, so we can reopen the issue and assign an owner to it.

[DasSkelett]

If someone wants to step up and work on this issue,
just let us know, so we can reopen the issue and assign an owner to it.

Someone already did, just nobody bothered to review it yet. #2071

So labelling it as "help wanted" doesn't make any sense, because apparently help is not wanted, otherwise someone from the team would've already looked at #2071

[AdSchellevis]

@DasSkelett our bot labels automatically, rules around plugins are quite simple. Every one has a maintainer, if he/she don't have time or spirit to do the work, they risk being expired in time (when not useable anymore for instance).

If someone else want to step up for a specific plugin, first try to ping the current maintainer and if that doesn't work you can always ask the core team to triage.

[fraenki]

@AdSchellevis Maybe the bot could be tuned to automatically request a review from the plugin maintainer when a PR is submitted? This would be useful. (I'd assume that otherwise PRs just get unnoticed because not everyone watched the plugins project.)

[DasSkelett]

@AdSchellevis the rules for plugins make sense.

For PR review, would it help if interested people test them and leave comments if it doesn't work/approve it if it works fine?

For the bot, you might want to skip issues with pending PRs, since the inactivity is pretty much explained by a solution being available but not merged yet.

[AdSchellevis]

@fraenki I think that's a good idea indeed, not sure if we can always track the maintainer automatically, but it's something to take under consideration when extending the bot's functionality in the future.