-
Notifications
You must be signed in to change notification settings - Fork 77
watchdog
Be aware that are multiple things that can all be referred to as a 'watchdog':
- the watchdog-counter in the silicium/SoC
- the watchdog-driver, loaded in the kernel
- the watchdog kernel module
- the watchdog-process in userland.
And on the boards hosting a nanopi, there is also an external watchdog. That was added because a bad power supply can wedge the entire SoC including the watchdog.
- Our kernels are configured with CONFIG_WATCHDOG_NOWAYOUT. This ensures that also accidentally stopping the watchdog-process will never lead to locked up system.
- In kernel version 4.13, a new feature was added: when the watchdog is already running when starting the kernel (as is our policy: enable watchdog in u-boot), it will ping the watchdog device while loading the kernel. This new feature is enabled by default. So, any kernels with version 4.13 or later need
# CONFIG_WATCHDOG_HANDLE_BOOT_ENABLED is not setin their config. And yes, thats a confusing way to disable a feature, but thats just how its done. - When initiation a reboot, the watchdog-process is stopped first. To make sure that a problem during a subsequent step in the reboot process can never lead to a system hang.
- The watchdog-process is stopped/killed under certain special conditions, to make sure the system resets (dbus daemon, connman, anything else?)
- The watchdog is started early, to watch over the system already during boot.
- But perhaps not for the CCGX? https://github.com/victronenergy/venus/issues/74
- What about nanopi? A20?
- The watchdog-process is configured to
- watch for max free memory as well as watch average system load.
- (in progress) check that connectivity to the VRM servers is up. https://github.com/victronenergy/venus/issues/287
- In case the watchdog-process sees a problem, it will make sure the system reboots and will also attempt to:
- write the reason to stop to a file, for VRM. See below
- write the process list to
/data/log/watchdog_processlist.txt - The watchdog-process will append a line to /data/wtmp in case of a repair.
Software originated, hence available on all machines:
| code | description | origin |
|---|---|---|
| -3 | tmp file already read, vrmlogger restarted!? | vrmlogger |
| -2 | Reading tmp file failed | vrmlogger |
| -1 | Reading watchdog register failed | get_boot_type.c |
| 29997 | Max load avg exceeded | watchdog-process |
| 30012 | watchdog-ENOMEM | watchdog-process |
| 30253 | watchdog-EMAXLOAD | watchdog-process |
Hardware related:
| code | description | ccgx | beaglebone | nanopi | CANvu500 |
|---|---|---|---|---|---|
| 0 | Hw without boottype support or old ccgx kernel | x | x | x | |
| 1 | Cold boot or reboot | x | |||
| 2 | Unreproducable reset on CCGX | x | |||
| 3 | Reset button | x | |||
| 4 | Cold boot | x | |||
| 5 | Reboot command | x | |||
| 17 | Watchdog reboot | x | x |
- For the nanopi we do use the watchdog in the SoC, but its driver doesn't export a boot type.
- https://github.com/victronenergy/venus/issues/10
When logged in a system that rebooted, you might well see this (TODO UPDATE THIS ONCE FIXED. THAT 53 IS WRONG):
root@ccgx:~# cat /tmp/last_boot_type
-3
53
root@ccgx:~# cat /tmp/last_boot_type.orig
30253
/data/log/watchdog_processlist.txt
And then there is also /data/wtmp, see man page.
In the VRM Portal we want to store the reason for a (re)boot, for diagnostics purposes.
If the watchdog-process decided to reboot, we are interested in its reason, and otherwise we are interested in the data in the microprocessor register.
The code we want to go to VRM is selected here, and written to /tmp/last_boot_type.
Vrmlogger reads the code, sends it, and then writes -3 in top of the file, meaning tmp file already read. Seeing that on VRM usually means that vrmlogger has restarted either on purpose or because it crashed. In case it can't read the file it will report -2 to vrm.
From watch_err.h in the software watchdog project (note, 255 -> 30253
#define EREBOOT 255 /* unconditional reboot (255 = -1 as unsigned 8-bit) */
#define ERESET 254 /* unconditional hard reset */
#define EMAXLOAD 253 /* load average too high */
#define ETOOHOT 252 /* too hot inside */
#define ENOLOAD 251 /* /proc/loadavg contains no data */
#define ENOCHANGE 250 /* file wasn't changed in the given interval */
#define EINVMEM 249 /* /proc/meminfo contains invalid data */
#define ECHKILL 248 /* child was killed by signal */
#define ETOOLONG 247 /* child didn't return in time */
#define EUSERVALUE 246 /* reserved for user error code */
#define EDONTKNOW 245 /* unknown, not "no error" (i.e. success) but implies test still running */
Getting a list from the database:
select valueEnum, nameEnum, count(*) as nr_of_sites, min(l.secondsAgo) / 60 as min_minutes_ago, max(l.secondsAgo) / 60 as max_minutes_ago from vwLastLogData l where idDataAttribute = 237 group by valueEnum;