The Agent/Gateway healthy condition is more reliable

[a-thaler]

Description
As part of #728 the user should be able to define alerts on the kyma module status. At the moment, there are situations where the telemetry module state is "unhealthy" by design while there is no unhealthiness. That happens mainly in upgrade procedures in node eviction situations where gracefully pods are getting replaced. The rollout is non-disruptive, but the module state indicates a disruption. and an alert will be fired.

The goal is to realize that situations differently so that the module gets "unhealthy" only in problematic situations where the user should react.

Hereby, the gateway/agentHealthy condition needs to be improved by not just checking if the pods are running, but checking for unhealthy conditions instead.

Criterias

• A regular update of the module should not result in a module state change
• A regular pod reschedulling caused for example by a node upgrade should not result in a module state change
• At definition of the first pipeline resource, the state should stay healthy till the agents are up or are in a bad state
• If an agent cannot come up (startup error, OOM, waiting for volume mount for too long), a problem gets indicated

Hints
The manager should not look anymore to the desired vs available replicas, instead it should check if a minimal amount of pods is available and if all pods are in a healthy state. We might have to watch the pods of the components additionally in order to react to pod status changes.

[a-thaler]

We agreed to give a better transparancy to the user by introducing a dedicated reason for the situation where not all pods are ready. As we interpret that situation as "healthy", the value will reflect a positive situation.

Agreed status:

• all desired pods are running -> healthy, 1
• Some pods are not ready but have no known bad state -> rolloutInProgress, 1
• a pod is in a known bad state or not scheduled at all -> unhealthy, 0

[rakesh-garimella]

The status was implemented as in the comment above.