distsql: remove possibility of inbound stream timeout errors

[asubiotto]

We currently set up a flow by sending out n SetupFlowRequests to all participating nodes in a query. These then set up any necessary connections between them by having outboxes send a FlowStream RPC to a destination node. These FlowStream RPCs wait up to a timeout for the corresponding flow to be registered on the destination node. Note that a SetupFlowRequest does not actually call RegisterFlow. That only happens once Start is called.

We have run into a bunch of issues with streams timing out because these two events don't line up within the timeout. I think we could get rid of a lot of these cases by having SetupFlowRequests include the FlowSpecs of all downstream nodes. The FlowStream requests would use these and could be made responsible to set up the flows on remote nodes actively instead of waiting for a SetupFlowRequest if the flow isn't registered.

More details to come.

[jordanlewis]

@asubiotto is this still a thing you want to do?

[asubiotto]

I think it would be interesting to explore, but don't think it's a high priority.

[asubiotto]

I explored this but don't think we want to go down this road at this time. There is essentially only one FlowSetupRequest that needs to be sent to any given node, but with this approach we'll introduce a lot of redundancy (essentially copying the same FlowSetupRequest to each outbox in a flow). This issue was prompted by streams timing out under load, which is something that @ajwerner's work will probably obviate so it seems better to keep the current, simple, status quo than try going down a path without many concrete benefits.