release-22.1: flowinfra: cancel remote flows when node is drained

[yuzefovich]

Backport 1/1 commits from #82752.

/cc @cockroachdb/release

---

This commit fixes an oversight in the draining process of the DistSQL
flows. Previously, it was possible for some flows to keep on running
even after server.shutdown.query_wait has passed (which acts as
a grace period to allow queries to complete). This only affects the
distributed queries since local queries are already canceled when the
connections to the node being drained are interrupted.

This commit makes it so that the flow registry actively cancels all
still running flows after the query wait grace period. This is done by
canceling the context of the flow. As a result, distributed queries that
have flows on the node being drained now will result in an error
(previously, they could stall the draining process until they would
complete).

Additionally, this commit fixes an oversight introduced in
yuzefovich@5ff1974 so that all flows (except for
fully-local queries) get registered with the flow registry. This matters
for remote flows that don't have any inbound connections (e.g.
SELECT count(*) query or a CDC flow) which would previously by-pass
the flow registry altogether.

Since this commit is a backport on an older release branch, we gate the
new behavior with a cluster setting that disables the new behavior by
default.

Release note (bug fix): When a CockroachDB node is being drained, all
queries that are still running on that node are now forcefully canceled
after waiting the server.shutdown.query_wait period if the newly-added
cluster setting sql.distsql.drain.cancel_after_wait.enabled is set to
true (it is false by default).

Release justification: bug fix.

[blathers-crl]

Thanks for opening a backport.

Please check the backport criteria before merging:

• Patches should only be created for serious issues or test-only changes.
• Patches should not break backwards-compatibility.
• Patches should change as little code as possible.
• Patches should not change on-disk formats or node communication protocols.
• Patches should not add new functionality.
• Patches must not add, edit, or otherwise modify cluster versions; or add version gates.

If some of the basic criteria cannot be satisfied, ensure that the exceptional criteria are satisfied within.

• There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
• The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
• New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters.
• The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.

Add a brief release justification to the body of your PR to justify this backport.

Some other things to consider:

• What did we do to ensure that a user that doesn’t know & care about this backport, has no idea that it happened?
• Will this work in a cluster of mixed patch versions? Did we test that?
• If a user upgrades a patch version, uses this feature, and then downgrades, what happens?

[cockroach-teamcity]

This change is 

[cucaroach]

I know this is a backport but do we not have any tests where cancelStillRunning is true?

Reviewed 7 of 7 files at r1, all commit messages.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @mgartner)

[yuzefovich]

Yeah, we don't. I guess I don't have a good excuse for not adding the tests in #82752 other than not really considering that and the change being pretty straightforward 🤷‍♂️ If you think it's worth it, I'll write something up on master and will backport.