[Heartbeat] New state created immediately after each monitor edit

[andrewvc]

The current states implementation has a bug where the first check for a given monitor will immediately transition after an edit. Editing should not cause this

This is a low priority bug since users do not yet use this feature.

[elasticmachine]

Pinging @elastic/uptime (Team:Uptime)

[andrewvc]

I think the cause of is that we can't reliably connect to ES to read prior states, so when that work is complete it should be fixed. The reason the states are lost on edits is that heartbeat (or rather libbeat to be precise) has no notion of a config update, just stop and start, so the monitor is stopped, its in memory state is lost, then it is started.

This isn't really so much a problem for ES since we'll at least have the last result in ES.

I think our options are:

1. Add a caching layer, where rather than query ES we get data from an in memory cache that includes recently stopped monitors
2. Use the local FS to store states as the default impl instead of NilStatesLoader, kinda like caching, but also solves the issue of what happens when users can't connect to ES (so long as they have a stable FS).
3. Wait for the ES loader to work properly

I do have some affinity for the caching idea, because one issue today is that ES may be slightly out of date if stuff is stuck in the beats pipeline. I think I'll probably proceed along those lines.

[emilioalvap]

Closed via #33405 and #91391. Tested working on 8.5.2

[justinkambic]

Post-FF Testing LGTM