Update disk check

[bernardjkim]

Description

This PR updates the disk check and addresses items 1-3 of gravitational/gravity#1662. The disk/storage check will now report a warning or a critical probe at different thresholds. The default is set to 80% disk usage for a warning probe and 90% disk usage for a critical probe.

Linked tickets and PRs

• Updates Split disk watermark check into low and high gravity#1662

Testing done

Testing done on version 5.5.47. fallocate is a quick way to alloate disk space for a file.

Verify warning when disk usage > 80%

[vagrant@node-1 gravity]$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
[...]
/dev/sdb1       25670884 20356768   3987064  84% /var/lib/gravity

[vagrant@node-1 gravity]$ sudo gravity status
[...]
Cluster nodes:
    Masters:
        * node-1 (172.28.128.101, node)
            Status:             healthy
            [!]                 disk utilization on /var/lib/gravity exceeds 80 percent (4.1GB is available out of 26GB), see https://gravitational.com/telekube/docs/cluster/#garbage-collection
            Remote access:      online
[...]

Verify critical when disk usage > 90%

[vagrant@node-1 gravity]$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
[...]
/dev/sdb1       25670884 22229512   2114320  92% /var/lib/gravity

[vagrant@node-1 gravity]$ sudo gravity status
[...]
Cluster nodes:
    Masters:
        * node-1 (172.28.128.101, node)
            Status:             degraded
            [×]                 disk utilization on /var/lib/gravity exceeds 90 percent (2.2GB is available out of 26GB), see https://gravitational.com/telekube/docs/cluster/#garbage-collection
            Remote access:      online
[...]

[r0mant]

I'd move this constants alongside the other one (the low watermark), I think it's somewhere in lib/constants.

[bernardjkim]

A watermark constant was not previously defined in satellite. The constant is defined in planet and passed to the storage checker.

[r0mant]

I'd just have 2 separate constants TBH for soft/hard limit. Is there any particular reason you wanted to "tie" them one to another this way?

[bernardjkim]

We want the watermark to be configurable/overrideable in emergency situations. If the values are separately configurable, we might run into incorrect behavior if the critical watermark is set to a lower value than the warning watermark. Having them tied together will make sure we don't run into that situation.

[r0mant]

This scenario can be just checked during checker initialization, right? In Check(), as we usually do. And just return an error if that's the case.

[bernardjkim]

Yeah, that works. I guess it'd be nicer to have separate control over the two threshold values.

[a-palchikov]

This should be also taken into account: gravitational/gravity#1748