[E2E] Agent recovers gracefully when upgrade is interrupted unexpectedly

[ycombinator]

Extracted from #2176:

Test that the agent reliably restarts with the current version when an upgrade to a new version is interrupted unexpectedly, e.g. restarting the Agent while downloading or right before switching the symlink and ensuring that the pre-upgrade Agent version is the one that comes up.

On Linux there is very likely a window here where failure is not recoverable, see analysis in #4834.

The reason release() not making a system call is interesting is because this means that the watcher is still child of the agent process, meaning it dies when the agent dies. This cannot be changed without a system call happening on Unix. This means there is an opportunity where the agent is dead and the watcher is also dead, unable to roll back. I think we may need to use something like https://github.com/sevlyar/go-daemon?tab=readme-ov-file#how-it-works to fix this. I suspect our existing test for this case isn't precise enough. The watcher from the agent starting the upgrade would need to be killed, and the agent we upgraded to would have to exit before it launched the watcher. This is not how it works today. I don't think this PR makes anything any worse but it doesn't fix this problem.

[cmacknz]

Updated the description, I think there is an actual bug here on Linux our tests aren't finding.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)