Server checkers works incorrectly when s8k/os workspaces are recovered

[sleshchenko]

Description

Server checkers works incorrectly when s8k/os workspaces are recovered.
It is needed to fix them and implement the following approach:

When tomcat is started recover all the workspaces.

• If a recovered workspace is RUNNING - start servers checkers immediately.

Listen to runtime status changes and:

• start server checkers when workspace becomes RUNNING (it is possible when old Che Server instance was starting a workspace).
• stop server checkers when workspace becomes STOPPED.

Reproduction Steps

1. Start workspace A and wait until workspace A will be started.
2. Start workspace B (it should be configured incorrectly to fail, like installer with broken script) and redeploy Che Server with Rolling Update.
3. Mannually stop wsagent in workspace A. Wsagent will still be considered as RUNNING.
4. Check updated Che Server logs, there will be error messages that unable to update server status for workspaec B.

OS and version:

Diagnostics:

[sleshchenko]

Implemented starting of servers checkers for all RUNNING recovered workspaces.

Is not implemented the following subtasks:

1. Starting server checkers when workspace becomes RUNNING after recovering (it is possible when old Che Server instance was starting a workspace). Note that it will affect a user only if there is STARTING workspace during Rolling Update and one of the following workspace servers failed: exec-agent, terminal, ws-agent. Created separated issue for that Servers checkers won't be started if a workspace is started by another Che Server instance #9502.
2. Stop server checkers when workspace becomes STOPPED. Note that it is not an issue for Rolling Update, but it is actual for replications (when there are more than one active Che Server at the same time).