release-23.2: server: option to wait for default_target_cluster on connect

[blathers-crl]

Backport 6/6 commits from #113687 on behalf of @stevendanna.

/cc @cockroachdb/release

---

Virtual cluster startup is asynchronous. As a result, it can be hard to
orchestrate bringing a virtual cluster up since the client needs to
retry until the cluster is available.

This makes it a little easier by optionally allowing connections to
the default_target_cluster to be held until the cluster is online.

As a result, a user who does:

ALTER VIRTUAL CLUSTER app START SERVICE SHARED
SET CLUSTER SETTING server.controller.default_target_cluster = 'app'

Should be able to immediately connect to the same node without having
to handle retries while the virtual cluster starts up.

Note that we can extend this to wait for any
tenant. The restriction on only waiting on the default_target_cluster
is that we need to thread a little state into the server controller in
order to avoid pitfalls such as holding a lot of connections open for
longer than needed for a tenant that will never become available
because it doesn't exist.

In a future PR, we will set this option by default in the replication-source and replication-target
configuration profiles.

Built on #113666

Informs #111637

Release note: None

---

Release justification: Usability fix for physical replication, making orchestration mistakes less likely.

[blathers-crl]

Thanks for opening a backport.

Please check the backport criteria before merging:

• Backports should only be created for serious
  issues or test-only changes.
• Backports should not break backwards-compatibility.
• Backports should change as little code as possible.
• Backports should not change on-disk formats or node communication protocols.
• Backports should not add new functionality (except as defined
  here).
• Backports must not add, edit, or otherwise modify cluster versions; or add version gates.
• All backports must be reviewed by the owning areas TL and one additional
  TL. For more information as to how that review should be conducted, please consult the backport
  policy.

If your backport adds new functionality, please ensure that the following additional criteria are satisfied:

• There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
• The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
• New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters. State changes must be further protected such that nodes running old binaries will not be negatively impacted by the new state (with a mixed version test added).
• The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.
• Your backport must be accompanied by a post to the appropriate Slack
  channel (#db-backports-point-releases or #db-backports-XX-X-release) for awareness and discussion.

Also, please add a brief release justification to the body of your PR to justify this
backport.

[cockroach-teamcity]

This change is 

[stevendanna]

This does add new functionality. But:

• The functionality should only impact users of shared-process multi-tenancy who have set: server.controller.default_target_cluster.
• There is a clear early-exit for the system tenant so this should not inadvertently affect existing single-tenant deployments:

https://github.com/cockroachdb/cockroach/pull/113933/files#diff-5012b40344254902c8506a506c562ae80ed3c1b5a8e033745297a5ba0497d06dR95-R99

It is likely not the minimal change; however, I'm wary of having different versions of this on master vs 23.2. If we are unhappy with the size of this change, I recommend we simply forgo the backport and live with the orchestration problem.