swarm/: NetworkBehaviour/ConnectionHandler cross-communication

[mxinden]

Description

Today there is no way to send generic local and remote node information from one NetworkBehaviour to another NetworkBehaviour, apart from NetworkBehaviour::inject_new_external_addr.

Examples of such generic information could be:

• Supported protocols by the local peer.
• Supported protocols of a remote peer.
• Addresses of a remote peer.

Motivation

One example use-case is the exchange of information betweenlibp2p-identify and libp2p-kad to support Kademlia Client Mode (see #2032).

1. libp2p-kad needs to know whether a remote peer advertises /ipfs/kad/1.0.0 via the identify protocol to decide whether to include the remote peer in its routing table or not.
2. libp2p-identify needs to know whether the local node runs in client or in server mode, i.e. supports the /ipfs/kad/1.0.0 or not.

See #2521 which is currently blocked on this.

Proposal

Below is an initial action plan proposal based on a meeting with @MarcoPolo.

1. Proof of concept 1 (communicating remote metadata across behaviours)
   • Extend NetworkBehaviourAction with NetworkBehaviourAction::ReportRemoteMetaData, containing a delta update.
   • Add NetworkBehaviourAction::inject_remote_metadata.
   • Write a Select NetworkBehaviour which takes two NetworkBehaviour implementations and passes ReportRemoteMetaData emitted by one behaviour to the other via inject_remote_metadata.
   • Showcase how libp2p-kad can learn supported protocols of remote peers from libp2p-identify.
2. Proof of concept 2 (communicating local metadata across behaviours)
   • Extend NetworkBehaviourAction with NetworkBehaviourAction::ReportSupportedProtocol which each NetworkBehaviour emits when it supports a protocol as a listener and which is passed to other NetworkBehaviours via the Select NetworkBehaviour.
   • Showcase how remote peers can learn that libp2p-kad moved from client mode into server mode via libp2p-identify.
3. In case proof of concepts looks good, don't merge but consolidate inject_* methods into single inject_event method with a single enum. Motivation here is to not have yet another inject_*** method on NetworkBehaviour.
4. Adjust proof of concepts, implement Select behaviour logic in libp2p-swarm-derive and merge.
5. Add method on Swarm (e.g. add_remote_metadata) which would call NetworkBehaviour::inject_event(Event::RemoteMetaData) on the root NetworkBehaviour.
6. Design AddressBook type which each NetworkBehaviour can instantiate and pass a reference from inject_event into to track peer metadata.

Open questions

Are you planning to do it yourself in a pull request?

No, or at least not any time soon.

[thomaseizinger]

Supported protocols are connection-specific so this should likely be reported on a connection-level by ConnectionHandlers and not NetworkBehaviours. Plus the connection is already tied to a peer so the swarm will simply "know" what we are talking about.

I think there is also a nice symmetry here: A peer changing its supported protocols only concerns that one connection so we should only tell that one ConnectionHandler about it. Whether that is reported up to the NetworkBehaviour is protocol-specific.

[mxinden]

Supported protocols are connection-specific so this should likely be reported on a connection-level by ConnectionHandlers and not NetworkBehaviours.

I have never thought about it this way. Thus far in libp2p-kad I was planning to include a peer in our routing table if it advertises the Kademlia protocol on any of its connection. I guess we could include the peer under the address of the connection only.

👍 I think this is worth exploring.

[thomaseizinger]

Supported protocols are connection-specific so this should likely be reported on a connection-level by ConnectionHandlers and not NetworkBehaviours.

I have never thought about it this way. Thus far in libp2p-kad I was planning to include a peer in our routing table if it advertises the Kademlia protocol on any of its connection. I guess we could include the peer under the address of the connection only.

👍 I think this is worth exploring.

I just realized that the above may be too rust-libp2p specific. In here, protocols are connection-specific (because bound to a ConnectionHandler). This may not be true for other implementations?

Going back to the specific kademlia example: In theory, a node could be publicly reachable on one transport (i.e. QUIC) but not on another (i.e. TCP) so each ConnectionHandler should probably decide for itself whether they underlying transport's address gets included the routing table or not?

[thomaseizinger]

Supported protocols are connection-specific so this should likely be reported on a connection-level by ConnectionHandlers and not NetworkBehaviours.

I have never thought about it this way. Thus far in libp2p-kad I was planning to include a peer in our routing table if it advertises the Kademlia protocol on any of its connection. I guess we could include the peer under the address of the connection only.

👍 I think this is worth exploring.

I just realized that the above may be too rust-libp2p specific. In here, protocols are connection-specific (because bound to a ConnectionHandler). This may not be true for other implementations?

Do we have any recommendations in the specs for this? cc @MarcoPolo

Is a node expected to report (and support) the same set of protocols on each connection?

[mxinden]

Do we have any recommendations in the specs for this? cc @MarcoPolo

I don't think this is specified anywhere.

Is a node expected to report (and support) the same set of protocols on each connection?

That was my mind-set thus far, with the exception of relayed connections.

All that said, I am in favor of exploring the approach you propose above.

[thomaseizinger]

Do we have any recommendations in the specs for this? cc @MarcoPolo

I don't think this is specified anywhere.

I opened an issue to discuss this: libp2p/specs#525

Is a node expected to report (and support) the same set of protocols on each connection?

That was my mind-set thus far, with the exception of relayed connections.

All that said, I am in favor of exploring the approach you propose above.

Sounds good, I'll work on this next after finishing up generic connection management.

[thomaseizinger]

I think it would be nice to have #2831 for this.

[thomaseizinger]

I've put some more thought into this after looking at the code:

1. We already call ConnectionHandler::listen_protocol for each stream upgrade. At this point, we know which protocols we support on this connection.
2. We can cache this list and diff it on every upgrade. If we find a change, we can notify the connection.
3. Implementations of ConnectionHandler that want to change their supported protocols would have to do so in other "callbacks" to the ConnectionHandler by adjusting state internally and emitting a different set of protocols on the next call to listen_protocol.

[thomaseizinger]

In #3579, an idea came up that touches on this issue. In the same way that #3651 immediately re-injects all protocols gathered from the ConnectionHandler as a list, we could add a command to NetworkBehaviour for reporting a peers address which would immediately get re-injected into the behaviour (and thus down the behaviour tree).

This would allow us to build a standalone address book component that can learn about the addresses of peers from other behaviours such as identify, kademlia or gossipsub. This could make it obsolete for each of these behaviours to participate in storing and emitting addresses upon a new peer being dialed.

[thomaseizinger]

This would allow us to build a standalone address book component that can learn about the addresses of peers from other behaviours such as identify, kademlia or gossipsub. This could make it obsolete for each of these behaviours to participate in storing and emitting addresses upon a new peer being dialed.

Alternatively, we could add a first-class address book to Swarm. What worries me a bit about that idea are requirements like persistent address books or async resolving of addresses. At least the former we wouldn't need to deal with if an address book is just another NetworkBehaviour.

[thomaseizinger]

With #3651, we've merged the necessary support for this to enable kademlia client mode. More importantly, this also sets a direction for how we want to resolve similar problems. In particular, the design to follow is to emit an event and immediately inject it into the component again.

Having this concept in place should make it straight-forward to implement the support for other solutions once the need arises.