elastic-agent: HEALTHY status fluctuates for fleet-server

[mtojek]

Spotted in elastic/integrations#950

We tried to use elastic-agent status as healthcheck for the fleet server, but apparently the stack initialization fails due to unstable HEALTHY status (suspected by @blakerouse ).

We would like to use the status command instead of /api/status, but this issue seems to be a blocker.

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[blakerouse]

@ph I think we should move this to 7.14. Being that it will take some investigating on the best way to fix this issue and it's not critical for the 7.13 release.

[ph]

Added to 7.14 and to the meta issue.

[urso]

@ph @blakerouse Do we have an update for this issue?

[ph]

Could this be similar to issue described in #25940 ?

[blakerouse]

I don't have an update for this issue. We need to determine how we want to improve the status command.

An approach to providing stability to this command would be to add debouce to healthy reporting. Basically requiring that healthy be reported for an X amount of time before actually saying internally that Elastic Agent is healthy.

[ruflin]

If I remember correctly, one issue in the past was that Elastic Agent was healthy but then goes to unhealthy state again when updating the policy. If this is the case, I'm not sure this is expected as the Elastic Agent itself is still healthy. Something similar was the case during the fleet-server self enrollment. What is our expected behaviour if the health status of the Elastic Agent if a subprocess is not healthy?

[blakerouse]

If a subprocess is not healthy then Elastic Agent should not be healthy. When updating policy the Elastic Agent does transition away from healthy to configuring, which is correct state for it. The issue is more that the Elastic Agent is reporting healthy when it should not be healthy, not the opposite direction.

[EricDavisX]

@mtojek do you have an easy way to reproduce this... maybe re-doing the change as you originally tried? or @blakerouse do you need that at this point?

[mtojek]

I think there is nothing special to reproduce. It's just interpretation of the behavior that Blake described above.

[EricDavisX]

I'm doing follow-ups on open issue for 7.14 - is this still under review / possible merge for 7.14? If we aren't attempting it, then please update the label to 7.15 or beyond and put it to backlog from 'iteration' as well. Thank you. @blakerouse

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!