[LW] Add additional fail-safe mechanisms around stored state

[Jolyon-S]

Goals (and why):
We've hit several memory leaks in the past within the lock watch code path. The aim here is to not only prevent these from taking down a service, but also to automatically surface them. Instead of adding metrics for each of these, they instead cause the cache to fall back. This in turn increments the metric for cache fallbacks, which we can then monitor or create alerts from.

Implementation Description (bullets):
Throw SafeIllegalStateException if internal structures exceed a constant threshold. In practice, these structures should be approximately proportional in size to the number of open transactions, which will almost never exceed 5k, and thus 20k is a very safe limit.

Note that any runtime exception is caught in the resilient cache and rewrapped as a retryable exception after the fallback has been selected.

Testing (What was existing testing like? What have you done to improve it?):
No tests for these bounds.

Concerns (what feedback would you like?):
Should we just take the $$$ hit and add metrics? My instincts say no, since we likely won't need the metrics unless there is a real issue.

Where should we start reviewing?:
Diff

Priority (whenever / two weeks / yesterday):
This/next week

[changelog-app]

Generate changelog in changelog/@unreleased

Type

• Feature
• Improvement
• Fix
• Break
• Deprecation
• Manual task
• Migration

Description

Add additional failsafe mechanisms around the size of lock watch internal state - this will cause the cache to fallback if it exceeds a certain size.

Check the box to generate changelog(s)

• Generate changelog entry

[gmaretic]

Looks good, but I'd add a bit more info to the log lines

[gmaretic]

same as above