Catch exception when string sent as metric value

[nmuesch]

What does this PR do?

Supersedes - #2274

Catches when a metric is not a float and logs a warning to the status page. Includes test

Motivation

The current behavior is that when a non numeric value is submitted as a metric, the check will completely fail and stop running. As an example, if a single postgres custom query starts returning a string instead of a number, this will cause the whole postgres check to stop running, causing ALL monitors over postgres to trigger. Instead, this makes it clear for the reason a metric stopped being submitted and also lets other, possibly mission critical metrics continue to be submitted.

Performance:
Worst case, we previously throw an exception and the check stops working, now we throw an exception and the check continues. This will incur a performance impact, but with the benefit of continuing to receive available metrics

In the best case, we have an extra try around something we wouldn't otherwise have had:

python -m "try:
    float(6)
except ValueError:
    pass"

returns:

10000000 loops, best of 3: 0.117 usec per loop

And in the existing functionality:

python -m timeit "float(6)"

returns:

10000000 loops, best of 3: 0.103 usec per loop

Ultimately a 0.01 usec difference with the gain of getting all other available metrics.

Review checklist

• PR has a meaningful title or PR has the no-changelog label attached
• Feature or bugfix has tests
• Git history is clean
• If PR impacts documentation, docs team has been notified or an issue has been opened on the documentation repo

Additional Notes

This adheres to the similar case where, if a single endpoint isn't available for an application, we continue retrieving available metrics.

[codecov-io]

Codecov Report

Merging #2293 into master will decrease coverage by 22.07%.
The diff coverage is 77.77%.

@@             Coverage Diff             @@
##           master    #2293       +/-   ##
===========================================
- Coverage   85.56%   63.49%   -22.08%     
===========================================
  Files         184       36      -148     
  Lines       12508     1915    -10593     
  Branches     1319      333      -986     
===========================================
- Hits        10703     1216     -9487     
+ Misses       1427      617      -810     
+ Partials      378       82      -296

[zippolyte]

LGTM

[masci]

👍 for avoiding false alerts

[ofek]

👍 Though, don't we want to retain the behavior of checks failing? Bad input is a hard error and making it just a log entry will obfuscate this for support when metrics don't show up.