Ensure we don't miss the resolution of an access request

[espadolini]

Right now tsh fires up a watcher to wait for the access request to be resolved and then creates the request; however it doesn't wait for the watcher to be ready, and so it's possible to miss the resolution of the request (be it approval, denial or removal) and we get a hang.

This makes it so that we wait for the watcher to be ready before proceeding with the request creation. In addition, we create the watcher on the root cluster rather than whatever cluster we're connected to.

Closes #9003 and #9244.

[espadolini]

How should I go about adding a test for this? #9003 has some instructions on how to replicate it, but it could also be tested by artificially increasing the time it takes for the watcher to be ready in the code.

[russjones]

@fspmarshall Any suggestions for @espadolini on how to write tests for this?

[espadolini]

After some minor refactoring, the watcher now waits for access request updates on the root cluster, rather than whatever cluster we're currently connected to.

[espadolini]

I added a test for the "access request while logged into a leaf cluster" scenario, it fails on master and passes after the changes in this PR. (for #9244)

I'm still not sure how to make a specific test for the spurious hangs of #9003; the test I just added does go through the codepath, so if we get a regression and the bug resurfaces we might eventually hit it in our CI runs anyway.

[fspmarshall]

I'm still not sure how to make a specific test for the spurious hangs

@espadolini Since this update removes the cause of the spurious hangs, I don't think there is a reasonable way to try to replicate them in a test. As you said, we cover the codepath, so if a new spurious hang is introduced, hopefully we'll catch it.

[russjones]

@espadolini Please backport this to branch/v8 and branch/v7. You can create backport PRs now, but don't merge until next week.