Nwaku maximum connections reached for wss clients

[jm-clius]

Problem

The nwaku-based wakuv2.prod and wakuv2.test fleet are currently reaching maximum connection capacity very soon after every restart. This seems to be due to scripted js-waku based clients connecting to the secure websocket connections available on those nodes and not releasing those connections. This means that other clients interested in connecting to the fleet nodes for any services are unable to do so as we've reached service capacity. The connections seem to be used for relay protocol.

This presents at least two problems:

• find a solution to managing/discouraging connection "hoggers", especially when providing scarce services to clients (such as store, filter, lightpush or even just wss listening ports).
• due to how these specific connections are used, the node resources (especially memory usage) has climbed very high and the fleet nodes have slowed down considerably.

Steps being taken

• An investigation into why these connections are causing such a high increase in memory usage: bug: possible memory leak nwaku#1662 and probably related to bug: messages being gossiped one hour later nwaku#1542 cc @Menduist
• Planned improvements to peer management, including considering expiring connections or discouraging multiple connections from the same IP cc @alrevuelta feat: blacklist peers sharing more than x IPs nwaku#1749
• Work with other clients, e.g. RAILGUN, on strategies around js-waku use of peer discovery and reliance on our fleet cc @waku-org/js-waku-developers
• Try to determine which content topic(s) (and therefore application) is most likely using these slots.

[fryorcraken]

Here are some action items for js-waku: waku-org/js-waku#1326

We are also working on using WebRTC for relay that should also decrease reliance on bootstrap nodes: waku-org/js-waku#1181

[fryorcraken]

Few notes:

1. Makes sense to de-prioritize Waku Relay connections over WSS, at least until [Epic] Waku Relay scalability in the Browser js-waku#905 is "done"
2. Make also sense to de-prioritize Waku Light Push, Store and Filter connections (over any transport) over Peer exchange connections, so that discovery has priority.
3. We may need to think/revisit how js-waku dials node. As it usually passes all protocols that are mounted. We do not mount relay if we don't intend to use so that's good news. However, we would usually mount and dial with peer exchange + light push + filter + store on the same connection. Maybe this should be changed? Maybe we should use two different connections for peer exchange and for light push + filter + store, WDYT @alrevuelta ?

[fryorcraken]

We have deployed a number of improvements:

• Waku fleet infra: Limit per IP websocket connections on wakuv2 fleets to 20 status-im/infra-nim-waku#70
• nwaku software (0.18.0.) feat(networking): prune peers from same ip beyond colocation limit nwaku#1765
• js-waku: [Connection Manager] Improve fallback mechanism when remote peer rejects connection js-waku#1326

We have improved peer management in nwaku in general (specifically by introducing scoring and pruning on IP colocation max). We're also working on a strategy for service-only peer connection slots.

Looking at the number of peers, we can see drastic changes on the 19 June 2023. This could be because of a number of a factors, but the gossipsub scoring and disincentivising IP colocation plays at least a role.

[mfw78]

Would the IP colocation limitation affect those behind NAT / CGNAT?

[alrevuelta]

@mfw78 Not really. The waku network aims to have a huge amount of nodes, so if one node drops your connection because of IP colocation (let say you run 10 devices under the same IP) thats fine, because there are many other nodes that you can connect to. In any case, forcing disconnection due to this it not only prevents attacks but also tries to enforce some diversity in the nodes that you are connected to.

[fryorcraken]

New issue logged: waku-org/nwaku#1831

[fryorcraken]

@jm-clius can this be closed? I have checked the grafana several times in the past few weeks and it seems much better.

[jm-clius]

Yes, we still have work planned for peer management, but the main issue has been addressed.