fix(servstate): use WaitDelay to avoid Command.Wait blocking on stdin/out/err

[benhoyt]

Use os.exec's Cmd.WaitDelay to ensure cmd.Wait() returns in a reasonable timeframe if the goroutines that cmd.Start() uses to copy stdin/out/err are blocked when copying due to a sub-subprocess holding onto them. Read more details about the issue in golang/go#23019 and the proposed solution (that was added in Go 1.20) in golang/go#50436.

This solves issue 149, where Patroni wasn't restarting properly even after a KILL signal was sent to it. I had originally mis-diagnosed this problem as an issue with Pebble not tracking the process tree of processes that daemonise and change their process group (which is still an issue, but is not causing this problem). The Patroni process wasn't being marked as finished at all due to being blocked on the cmd.Wait(). Patroni starts sub-processes and "forwards" stdin/out/err, so the copy goroutines block. Thankfully Go 1.20 introduced WaitDelay to allow you to easily work around this exact problem.

The fix itself is this one-liner:

s.cmd.WaitDelay = s.killDelay() // defaults to 5 seconds

This will really only be a problem for services, but we make the same change for exec and exec health checks as it won't hurt there either.

Also, as a drive-by, this PR also canonicalises some log messages: our style is to start with an uppercase letter (for logs, not errors) and to use "Cannot X" rather than "Error Xing".

Fixes #149.

[hpidcock]

We might need special handling of exec.ErrWaitDelay

[benhoyt]

@hpidcock No, I don't think we need to handle ErrWaitDelay specially. Normally pebble services are terminated with a signal, so will have a "terminated" exit code and that will be the error reported. ErrWaitDelay is only returned if the wait-delay kicks in and the service terminates successfully (with a zero exit code) -- but we do want an error in that case as it means the children haven't cleaned up I/O properly. Additionally, in reaper.WaitCommand we expect a non-success err from cmd.Wait() due to the reaper already waiting for PIDs, so we always get a syscall error from "wait" or "waitid".

[flotter]

Looks great. I have one concern reusing killDelay - please check if my understand makes sense or not.

There is one subtlety which this new feature may create in the future - since a service restart were never able to restart child process that promoted themselves outside the process group (or session), in the past that may have forced people to someone find ways to kill everything or restart. What we are now enabling is leaving a child behind running, and killing the file descriptors previously used to capture the output (is my understanding correct)? If this is true, a service restart will cause the new primary process to log output without having the stdout of the previous children included (This is assuming the service knows how to deal with reusing the running child, else it may even start another one)?

[benhoyt]

There is one subtlety which this new feature may create in the future - since a service restart were never able to restart child process that promoted themselves outside the process group (or session), in the past that may have forced people to someone find ways to kill everything or restart. What we are now enabling is leaving a child behind running, and killing the file descriptors previously used to capture the output (is my understanding correct)? If this is true, a service restart will cause the new primary process to log output without having the stdout of the previous children included (This is assuming the service knows how to deal with reusing the running child, else it may even start another one)?

@flotter Yeah, that understanding is correct. However, I think the behaviour in this PR is strictly better than what we have now, which is causing a deadlock, and causing the charm (or whatever's running Pebble) to not see that the service stopped at all. People that were working around may have to change (or remove!) their workaround, but I think that's okay. Also, I think it was only the data team using Patroni that this affected (at least, that we knew about) -- and they know about this fix.

[marco6]

Adding my 2 cents