Peer management enhancements

[peterbroadhurst]

What would you like to be added?

• Better thread management of state distribution backlog and retries
• Status information to users for "active peers"
• Clear code responsibility for managing the node-to-node transport definitions

Why is this needed?

See:

paladin/core/go/internal/statedistribution/state_distributer.go

Lines 115 to 116 in a06db21

	// TODO: This needs to be a worker per-peer - probably a whole state distributor per peer that can be swapped in/out.
	// Currently it only runs on startup, and pushes all state distributions from before the startup time into the distributor.

[peterbroadhurst]

Initial architecture plan:

• The responsibility is combined into transportmgr from statedistribution (which gets removed)
• There is an in-memory status + go-routine for every "active peer"
• This go-routine is responsible for completing state distributions
• This go-routine also processes requests to send things from a dedicated go channel to that peer
• transportmgr takes ownership of the payload spec for messages
• Only spec-defined messages can be sent/received
• A short in-memory retry is done by transport manager to handle very intermittent errors to minimize complexity in transport plugins
• Messages are still one-way fire&forget, but these short retries manage very short reconnect-style error scenarios
• Status (bytes sent, lifetime of active, etc.) on failures is available over JSON/RPC on transport_peers / transport_peerInfo
• The transport plugin interface is updated with an explicit lifecycle of activate/deactivate on peers, rather than just send
  

[peterbroadhurst]

Implementation is in #491