Peer Management: Connection and Disconnection

[fryorcraken]

Planned start date:
Due date:

Summary

In a browser environment, loss of connectivity may happen.

When it happens, this would impede the application to receive and send messages.

Strategies need to be designed to:

• detect a loss of connection
• Automated actions when resuming a connection
  • auto reconnect to remote peer
  • Auto trigger some protocols (e.g. Waku Store Retrieve Store Messages when resuming connectivity #252)
  • Renewing Filter subscription
  • Waiting for peer to be in mesh relay before considering the action live (eg waitForRemotePeer)
• cope with lack of connectivity when trying to use a protocol: retry strategy, feedback to the app and guideline on how to handle
  • Querying message using Waku Store: eg Peer not found despite using waitForRemotePeer #913
  • Trying to send a message with light push

Proposed Solutions

The result of this issue would be a mix of:

• Documentation to guide the developers in handling failure
• Strategy embedded in the protocols (e.g. retry)
• Helper library that can provide utility to developers (e.g. auto store query, auto filter renewal)

At this stage it is not sure what should go in waku core and what should be a library helper.

Acceptance Criteria

• Upon start-up, app dial a reasonable set of peers from peer discovery methods (no persistence, only dial 1 bootstrap peer)
• Upon start-up, if some peers are not reachable, app attempt fallback method (try 2nd bootstrap peer)
• Following above rules, when disconnected to a remote peer, attempts to reconnect
• Clear guidelines for developer to know when node is connected or disconnected Advanced docs for js-waku docs.waku.org#104 (comment)

Tasks

• setup manual controlled connection management for peers (some work being done here: feat!: ConnectionManager and KeepAliveManager #1135)
  • remove libp2p's autodial
  • dial peer on the basis of set requirements, redial if necessary, stop dialing after N attempts
  • connect to minimum bootstrap nodes (1) and give preference to other discovery mechanisms (peer-exchange)
  • dial all available bootstrap peers if no PX peers are dialable after N seconds
• Investigate the effect of connection loss -> tracked here: (re)investigate gauging peer disconnections with js-waku #1412
  • libp2p events w.r.t. connected/disconnected peers (Future Improvements for ConnectionManager #1189 (comment))
  • auto-connection to disconnected peers
  • bug: nwaku disconnects every 10 minutes nwaku#2116
• pingKeepAlive throws uncaught exception #1646
• Advanced docs for js-waku docs.waku.org#104 (comment)
• fix: don't dial discovered peers if have already been attempted dial #1657

Tasks Moved to Out of Scope

• Waku Filter subscription state after disconnection/re-connection
• Waku Relay state after disconnection/re-connection
  • investigate alternative to relay pings/why we need them #1349
• Usage of Waku Light Push during connection loss
• Usage of Waku Store during connection loss
• Review possible automated actions upon re-connection (based on learnings from above)
  • auto store query Peer not found despite using waitForRemotePeer #913
  • auto filter subscribe
  • Implement WAKU2-LIGHTPUSH #132
• multiple connections opened for the same peer #1459

Notes

libp2p/js-libp2p#744

RAID (Risks, Assumptions, Issues and Dependencies)

[fryorcraken]

Need to re-groom this issue to take peer discovery in account. Some draft notes:

    // createLightNode => 2 light push, 2 filter, 1 store
    // dns discovery: emits "bootstrap" nodes
    // catch "bootstrap" node -> peer exchange? connect to it
    // How do we know it has peer exchange? need to extend peerInfo to include `waku2` ENR field or full ENR
    // Already connected to a "bootstrap" node? new node, park it.
    // random order? probably at dns discovery level, because we know when we are "done".
    // Now we receive a `peer-exchange` node
    // -> Do we need this node? What capabilities does it have?
    // -> What ndes are we currently connected to?
    // -> connect, discard or save for later (add to peer store)
    // -> Do a filter query: make it lazy, no wait for remote peer
    // Peer management module: good defaults and configurable/swappable
    ```

[danisharora099]

Since this issue is a considerable overhaul, I'd like to be sure of my interpretation of the same.

I think the biggest one is:

* detect a loss of connection

For this, my interpretation is that libp2p's connection manager doesn't emit an event when a peer disconnects/streams are closed (on peer:disconnect) (ref: status-im/status-web#288 (comment)) and that this will either be something we'll be handling ourselves (similar to status-im/status-web#288 or xmtp/xmtp-js#128) or an upstream change to libp2p (cont of libp2p/js-libp2p#939 / libp2p/js-libp2p#744)
Please let me know if that's a fair interpretation.

Secondly, an explanation on the following, of how this fits with the initially described scope of the PR, will also be greatly appreciated.

Need to re-groom this issue to take peer discovery in account. Some draft notes:

    // createLightNode => 2 light push, 2 filter, 1 store
    // dns discovery: emits "bootstrap" nodes
    // catch "bootstrap" node -> peer exchange? connect to it
    // How do we know it has peer exchange? need to extend peerInfo to include `waku2` ENR field or full ENR
    // Already connected to a "bootstrap" node? new node, park it.
    // random order? probably at dns discovery level, because we know when we are "done".
    // Now we receive a `peer-exchange` node
    // -> Do we need this node? What capabilities does it have?
    // -> What ndes are we currently connected to?
    // -> connect, discard or save for later (add to peer store)
    // -> Do a filter query: make it lazy, no wait for remote peer
    // Peer management module: good defaults and configurable/swappable
    ```

Other than this, appreciate that the issue has been written very descriptively and will have a great impact!

cc @fryorcraken @felicio

[fryorcraken]

As mentioned. this is a considerable overhaul and would need to be tackled iteratively. Each iteration can increase the functionality/complexity of the manager while taking onboard feedback from dogfooding.

I suggest to focus the first iterations on using the peer discovery protocols dns discovery and peer exchange with the following logic:

1. when using createLightNode, apply default values of wanting 2 light push nodes, 2 filters nodes and 1 store node (user can override values).
2. DNS discovery bootstrap method is used, it discovers and emit nodes that are tagged as bootstrap
3. Connection Manager processes the bootstrap node
   i. Can connect to it and use it for lightpush/store/filter (we can do that now and later remove this logic)
   ii. Use peer exchange to discover more nodes
4. Connection Manager discards/saves for later other bootstrap nodes
5. peer exchange discover new nodes (using a bootstrap node), tags then as peer-exchange
6. peer-exchange node is discovered/emitted, connect to it (that's our second light push/filter node)
7. connect to more peer-exchange node to fulfill requirements set in (1) if needed.

(3) introduces an issue: the first node discovery by DNS discovery will always be the same given a given enr tree. Hence we may need to introduce some pseudo random shuffling logic. this should probably be done in the connection manager.

The final result above can already be split in several tasks/PRs.

Then, we can add error resilience on top of it, I'd recommend in the following order:

1. Fail to dial a node: handle this and try another node (easy as all located in connection manager)
2. protocol failed on a node (ie, light push query failed): disconnect node and try other node: this is harder because the protocol needs to feedback the failure to the manager which can then handle the disconnect/connection. Also the protocol should know whether the error is worthy a disconnection. This is not really feasible now (no differentiations in errors) but may be with the waku store protocol overhaul).
3. Node disconnects: as you mentioned, we need to have feedback from libp2p for that so it may be more difficult.

[danisharora099]

3. Connection Manager processes the bootstrap node

We do two things with the peer, is that what we mean by "process" here?

• Tag the peer within the PeerStore
• Dial

i. Can connect to it and use it for lightpush/store/filter (we can do that now and later remove this logic)

• why do we want to “later remove this logic”?

4. Connection Manager discards/saves for later other bootstrap nodes

• How does the ConnectionManager do that? Do we mean the PeerStore/ AddressManager?
• (later, making a note: we might also have to think about the edge cases like the addresses changing for these saved peers in the interim)

5. peer exchange discover new nodes (using a bootstrap node), tags then as peer-exchange
6. peer-exchange node is discovered/emitted, connect to it (that's our second light push/filter node)

• “that’s our second node” — this step is skipped if the initial node requirements are met, correct?
• the nodes are anyway connected to (unless ConnectionManager's limit is reached, in which case it will auto-prune)

1. Fail to dial a node: handle this and try another node (easy as all located in connection manager)

• how can we reproduce a non-dialable node?
• in the case the dial fails, the ConnectionManager/PeerStore should automatically ignore the node (from my understanding, with autoDial:true (which is used by default), all nodes are tried to connect to)

---

also, a lot of the process you describe, from my understanding, takes place implicitly by libp2p
for eg:

• discovery, tagging, dial attempts, etc are all handled implicitly

I'm not entirely sure what you mean by manually handling the above-described process

we might not need to manually handle/connect to discovered peers as libp2p automatically dials to discovered peers and handles pruning if necessary (upper connection limit on ConnectionManager) — we should just focus on discovering the most number of peers (ref: #1117)

Please let me know if I'm missing something

[fryorcraken]

3. Connection Manager processes the bootstrap node

We do two things with the peer, is that what we mean by "process" here?

* Tag the peer within the PeerStore

The tagging is done by the discovery service:

js-waku/packages/peer-exchange/src/waku_peer_exchange_discovery.ts

Line 92 in 0b08320

await this.components.peerStore.tagPeer(

* Dial

The dialling is done by the connection manager:

js-waku/packages/core/src/lib/waku.ts

Line 140 in 0b08320

libp2p.dial(peerId).catch((err) => {

We will have to make this trivial logic more complex in the future and have a proper ConnectionManager class/module.

By "process" I mean deciding whether to dial and dialing.

i. Can connect to it and use it for lightpush/store/filter (we can do that now and later remove this logic)

* why do we want to “later remove this logic”?

Because if all js-waku nodes connects to bootstrap nodes to be served (store/light push/filter), then the waku network will be centralized because it would only use the bootstrap nodes.
It would also mean that the bootstrap nodes would need to be high performing node to serve all js-waku nodes.

Instead, I propose for the bootstrap node to only be used for bootstrapping, ie, getting access to the network. meaning only used for discv5 and peer-exchange.

4. Connection Manager discards/saves for later other bootstrap nodes

* How does the `ConnectionManager` do that? Do we mean the `PeerStore`/  `AddressManager`?

I don't know. The node will be tagged as bootstrap and store in peerStore so I guess that's good enough.
Or maybe we'll have to also store the nodes in the ConnectionManager. We'll have to decide at implementation time if the peerStore gives us enough.

The ConnectManager is the new module that handles most of the logic described here.

* (later, making a note: we might also have to think about the edge cases like the addresses changing for these saved peers in the interim)

5. peer exchange discover new nodes (using a bootstrap node), tags then as peer-exchange
6. peer-exchange node is discovered/emitted, connect to it (that's our second light push/filter node)

* “that’s our second node” — this step is skipped if the initial node requirements are met, correct?

Yes. For the rest of the logic I assume the proposed default requirements are applied.

* the nodes are anyway connected to (unless ConnectionManager's limit is reached, in which case it will auto-prune)

We should automatically connect to all node we discover. Which is why we need a Connection Manager and why we need to implement this logic.

1. Fail to dial a node: handle this and try another node (easy as all located in connection manager)

* how can we reproduce a non-dialable node?

Pass an invalid multiaddr.

* in the case the dial fails, the ConnectionManager/PeerStore should automatically ignore the node (from my understanding, with `autoDial:true` (which is used by default), all nodes are tried to connect to)

We'll have to review in details the retry logic when it's time to implement.

also, a lot of the process you describe, from my understanding, takes place implicitly by libp2p for eg:

* discovery, tagging, dial attempts, etc are all handled implicitly

We should not dial every node we discover. This is why custom connection management is needed.

I'm not entirely sure what you mean by manually handling the above-described process

we might not need to manually handle/connect to discovered peers as libp2p automatically dials to discovered peers and handles pruning if necessary (upper connection limit on ConnectionManager) — we should just focus on discovering the most number of peers (ref: #1117)

Please let me know if I'm missing something

[danisharora099]

By "process" I mean deciding whether to dial and dialing.

How are we deducing whether to dial or not?
If there's a peer available other than bootstrap, we give it more preference and dial to it instead of the bootstrap peer - correct? (to increase decentralisation)

Because if all js-waku nodes connects to bootstrap nodes to be served (store/light push/filter), then the waku network will be centralized because it would only use the bootstrap nodes. It would also mean that the bootstrap nodes would need to be high performing node to serve all js-waku nodes.
That makes sense! We should prioritize nodes found by discovery mechanisms other the bootstrapping. Noted.

* the nodes are anyway connected to (unless ConnectionManager's limit is reached, in which case it will auto-prune)

We should automatically connect to all node we discover. Which is why we need a Connection Manager and why we need to implement this logic.

We should not dial every node we discover. This is why custom connection management is needed.

My interpretation so far:
Our connection priority should be something along the lines of:

• connect to only 1 bootstrap peer (as little as possible to avoid centralisation on bootstrap nodes)
• find more peers using other discovery mechanisms, via the bootstrap peer
• connect to all new peers found until requirements are fulfilled

is that a fair interpretation?

[danisharora099]

Also, as concluded from #1117 (comment), libp2p by default autodials.

Considering the scope of this PR, it's best to default autoDial to false considering we don't want to connect to all discovered bootstrap nodes immediately.

[fryorcraken]

My interpretation so far: Our connection priority should be something along the lines of:

* connect to only 1 bootstrap peer (as little as possible to avoid centralisation on bootstrap nodes)

* find more peers using other discovery mechanisms, via the bootstrap peer

* connect to all new peers found until requirements are fulfilled

is that a fair interpretation?

Yes.

[fryorcraken]

As discussed with @danisharora099 the first step was actually a focus on connections to node discovered, Need to update description to record this first step.

Next, would be exploratory work to understand what information we can get when a node is disconnected.

[fryorcraken]

Blocked by #1412

[danisharora099]

referring from #1412,

upon further investigation and taking into context previous findings, it's safe to conclude:

• connection:close: monitor a permanent connection close between the local & remote node (which should usually be triggered along with peer:disconnect for cases where we only have one connection with the remote node)
• peer:disconnect: monitor permanent disconnections with peers (this would imply not that no the underlying connection(s) have been permanently closed, and the only way to communicate with this peer again is to open a new connection/reconnect)
• pings will fail when there are temporary network degradations or reachability issues. this does not mean that the underlying connection has been closed.

---

to address part of the acceptance criteria from this milestone,
the principal question remains: how do we understand if the disconnection with the remote peer is deliberate, or simply due to network conditions (implying we want to initiate a reconnection)

as we concluded with #1403 (comment), js-waku redials automatically after the 10 mins mark, which is not a conscious disconnection and requires reconnection so perhaps the above question is enforced by default already?

cc @fryorcraken

[danisharora099]

Weekly Update

• achieved: investigated & closed (re)investigate gauging peer disconnections with js-waku #1412
• next: look into addressing deliberate vs accidental disconnections

[danisharora099]

the principal question remains: how do we understand if the disconnection with the remote peer is deliberate, or simply due to network conditions (implying we want to initiate a reconnection)

for this, we want to simply attempt reconnections

---

For library consumers,

For Filter,

• we should open a subscription, and also send recurring SUBSCRIBE_PING to the node to ensure we're maintaing active subscriptions

  • if we don't have an active subscription, we should reinitiate a subscription with the node

• in case of reconnection, SUBSCRIBER_PING should be used to check for active subscription on the service nodes' end

• write explicit tests to cover this hypothetical as much as possible Advanced docs for js-waku docs.waku.org#104 (comment)

• add something like manage your filter subscriptions to advanced js-waku docs Advanced docs for js-waku docs.waku.org#104 (comment)

[danisharora099]

Weekly Update

• achieved: reached a conclusion tackling deliberate vs accidental disconnections, PRs opened to handle Filter subscriptions on disconnection/reconnections, iterative fixes on addressing multiple dial attempts for same peer, fixes around keep alive pings
• next: getting reviews & merging these PRs which should enable us to close this epic 🥳

[danisharora099]

Weekly Update

• achieved: The Connection and Disconnection Peer Management epic has been closed