Waku Connect Playground nodes

[D4nte]

Can you please deploy a set of Waku nodes dedicated to the Waku Connect team. The nodes will be used as playground for:

• Projects interested in Waku / onboarded by Waku Connect to try out Waku/js-waku before they deploy their own node,
• Js-waku ad-hoc testing of specific behaviors in regards of discovery, connectivity, performance,
• Hackers during Hackathons with Waku Connect bounties, building dApps with js-waku.

Specs

Minimal specs at this stage as we only want to run relay. It'll also be a good opportunity to monitor waku nodes resources consumption.
The equivalent of a Digital Ocean's basic droplet should be good enough:

• 1 vCPU (I don't know the frequency, 1GHz?)
• 1 GB Ram
• 25GB SSD

Location/number

Provider does not matter.

• 3 nim-waku nodes around the world (EU/AM/AS)
• 1 go-waku node in Europe

No need for a prod/test. Just one set as described above. In terms of hostname it would make sense to have playground and waku in it.

Software setup

nim-waku

Same setup for all 3:

• websocket: yes
• no websockify
• no store

Other parameters can be the same than the nim-waku prod fleet.

go-waku

• websocket: yes
• no store
• yes relay
• yes lightpush
• yes filter

Others can be done as per the go-waku test fleet.

Deadline

We have a hackathon in March. I need to confirm the date and would prefer to have this done 2 weeks beforehand so that I have time to review. Let's say 21 Feb for now but I'll review the date to possible later once I know the hackathon date.

[jakubgs]

I extracted the nim-waku role into it's own repo - infra-role-nim-waku - and made several fixes:

• bc8bd3ff - drop unused nim_waku_light_node_enabled
• d3dccaf9 - drop unused nim_waku_bootstrap_fleet
• ef0aea42 - combine protocol config into nim_waku_protocols_enabled
• 91a84762 - nim_waku_websocket_cont_port to nim_waku_websock_port
• ed6b05fa - fix nim-waku container healthcheck
• b906edf6 - wait for RPC port to be available before calling
• 4ffcfc82 - readme: update websocket instructions

And I updated infra-nim-waku to use the new separate role repo: status-im/infra-nim-waku@9676e4e8

[jakubgs]

Also had to fix the reload of Consul service definitions: status-im/infra-role-consul-service@ac9796d0

[jakubgs]

Another fix to make websockify optional in Consul service definition: status-im/infra-role-nim-waku@cfe424c0

[jakubgs]

SSH access has been added, with sudo: 4c74a9e7

[jakubgs]

And deployed the nim-waku nodes: a3f68037

 > a all --become-user=admin -a 'docker ps'     
node-01.do-ams3.waku.connect | CHANGED | rc=0 >>
CONTAINER ID   NAMES      IMAGE                                CREATED              STATUS
c514d2788441   nim-waku   statusteam/nim-waku:deploy-v2-prod   About a minute ago   Up About a minute (healthy)
node-01.gc-us-central1-a.waku.connect | CHANGED | rc=0 >>
CONTAINER ID   NAMES      IMAGE                                CREATED              STATUS
8e71f8df3631   nim-waku   statusteam/nim-waku:deploy-v2-prod   About a minute ago   Up About a minute (healthy)
node-01.ac-cn-hongkong-c.waku.connect | CHANGED | rc=0 >>
CONTAINER ID   NAMES      IMAGE                                CREATED              STATUS
8ba34ca61622   nim-waku   statusteam/nim-waku:deploy-v2-prod   About a minute ago   Up About a minute (healthy)

[jakubgs]

I had to fix lack of websocket service definition: status-im/infra-role-nim-waku@bfbc9bf7

[jakubgs]

And added it to the status-fleets refresh script: https://github.com/status-im/infra-misc/commit/ca4cb328

 > c fleets.status.im | jq '.fleets."waku.connect"'
{
  "tcp/p2p/waku": {
    "node-01.ac-cn-hongkong-c.waku.connect": "/ip4/47.242.185.35/tcp/30303/p2p/16Uiu2HAm75XUMGev2Ti74G3wUzhyxCtbaDKVWzNwbq3tn5WfzRd4",
    "node-01.do-ams3.waku.connect": "/ip4/206.189.242.0/tcp/30303/p2p/16Uiu2HAm9VLETt1xBwDAwfKxj2XvAZDw73Bn4HQf11U26JGDxqZD",
    "node-01.gc-us-central1-a.waku.connect": "/ip4/34.122.196.134/tcp/30303/p2p/16Uiu2HAmMh9Jy2KzrLYHDmekyS6F7wB3F9TN6zbNBf8rCvUDcD8j"
  },
  "wss/p2p/waku": {
    "node-01.ac-cn-hongkong-c.waku.connect": "/dns4/node-01.ac-cn-hongkong-c.waku.connect.statusim.net/tcp/443/wss/p2p/16Uiu2HAm75XUMGev2Ti74G3wUzhyxCtbaDKVWzNwbq3tn5WfzRd4",
    "node-01.do-ams3.waku.connect": "/dns4/node-01.do-ams3.waku.connect.statusim.net/tcp/443/wss/p2p/16Uiu2HAm9VLETt1xBwDAwfKxj2XvAZDw73Bn4HQf11U26JGDxqZD",
    "node-01.gc-us-central1-a.waku.connect": "/dns4/node-01.gc-us-central1-a.waku.connect.statusim.net/tcp/443/wss/p2p/16Uiu2HAmMh9Jy2KzrLYHDmekyS6F7wB3F9TN6zbNBf8rCvUDcD8j"
  }
}

[jakubgs]

@D4nte I'm away half day today, and I probably won't have time to get the go-waku node up, but I'll get it tomorrow for sure.

Please test the ones we already have and tell me if I got anything wrong.

[D4nte]

@D4nte I'm away half day today, and I probably won't have time to get the go-waku node up, but I'll get it tomorrow for sure.

Please test the ones we already have and tell me if I got anything wrong.

Looks good I played with it via the tcp ports + ssh. I'll need to wait for waku-org/nwaku#836 to be released to confirm that native websocket work properly deployed.

[jakubgs]

Indeed, it appears waku-org/nwaku#836 was merged on 1st of February while the current deploy-v2-prod Docker image tag was built on 20th of January and is on 81f89b5b: https://ci.status.im/job/nim-waku/job/deploy-v2-prod/26/

@jm-clius When is this going to get released/deployed to prod?

[jm-clius]

Rough target for the release is end of next week. We have a bunch of critical things that need to be bundled. I'll create a tracking issue for the release soon.

[jakubgs]

That's great, thanks! If need be we can always deploy a non-prod image to the infra-waku-connect fleet.

[jakubgs]

I've decided to rename the nodes to match the type of Waku implementation used:

nim-01.ac-cn-hongkong-c.waku.connect
nim-01.do-ams3.waku.connect
nim-01.gc-us-central1-a.waku.connect

Change: cd568140

[jakubgs]

Deployed a host node for go-waku node:

• ad7e994d - add new go-01.do-ams3.waku.connect host
• 6d980850 - deploy go-waku node on the new host

It works:

[email protected]:~ % docker ps
CONTAINER ID   NAMES          IMAGE                            CREATED         STATUS
4f1ce82aa297   go-waku-node   statusteam/go-waku:deploy-test   5 minutes ago   Up 5 minutes

[jakubgs]

Small fix for Certbot certificate refresh config: 48742fa9

[D4nte]

That's great, thanks! If need be we can always deploy a non-prod image to the infra-waku-connect fleet.

Ah great, if that's the same to you @jakubgs, can we please deploy nim-waku's bb3e59454ea4e34ccd80ffe2dd5cca170e64366d on the fleet please? Do you prefer an issue for that?

[D4nte]

Do you have the details (peer id) of the go-waku node? I found the ip but I don't know where to look for the logs.

[jakubgs]

Current master appears to fail with Unrecognized option 'discovery'.

[jakubgs]

Doesn't seem to be an existing flag anymore:

[email protected]:~ % d run --rm -it statusteam/nim-waku:deploy-waku-connect --help | grep discovery 
 --listen-address          Listening address for LibP2P (and Discovery v5,
 --dns-discovery           Enable discovering nodes via DNS [=false].
 --dns-discovery-url       URL for DNS node list in format
 --dns-discovery-name-server  DNS name server IPs to query. Argument may be
 --discv5-discovery        Enable discovering nodes via Node Discovery v5
 --discv5-udp-port         Listening UDP port for Node Discovery v5.
 --discv5-enr-auto-update  Discovery can automatically update its ENR with

@jm-clius should I just use the v5? Or drop the flag. @D4nte what do you think?

[jakubgs]

I've configured nim-waku builds for the fleet under deploy-waku-connect tag:
https://ci.status.im/job/nim-waku/job/deploy-waku-connect/

But not sure what about the discovery part.

[D4nte]

Doesn't seem to be an existing flag anymore:

[email protected]:~ % d run --rm -it statusteam/nim-waku:deploy-waku-connect --help | grep discovery 
 --listen-address          Listening address for LibP2P (and Discovery v5,
 --dns-discovery           Enable discovering nodes via DNS [=false].
 --dns-discovery-url       URL for DNS node list in format
 --dns-discovery-name-server  DNS name server IPs to query. Argument may be
 --discv5-discovery        Enable discovering nodes via Node Discovery v5
 --discv5-udp-port         Listening UDP port for Node Discovery v5.
 --discv5-enr-auto-update  Discovery can automatically update its ENR with

@jm-clius should I just use the v5? Or drop the flag. @D4nte what do you think?

What was the discovery option? What argument did you pass?

I think I noticed you used --staticnode. It's good enough for now. Once I have done some testing, I will open an issue to setup a DNS Discovery name for the Waku Connect node that we'll pass using --dns-discovery-url.

[jm-clius]

@jm-clius should I just use the v5? Or drop the flag. @D4nte what do you think?

Yeah, discovery was a v1 option and is not available in v2. I agree that --staticnode is good enough for now, followed by DNS discovery.

[jakubgs]

Ok, I see what's happening, that's because we check if it's v1 or v2 based on container image tag:
https://github.com/status-im/infra-role-nim-waku/blob/5d94f2f2ba8251af2837b4e4b6d05d17ce9e23a5/templates/docker-compose.yml.j2#L35
But when I renamed the container image name to use deploy-waku-connect tag that caused the role to use v1 flags.

[jakubgs]

Fixed:

• status-im/infra-role-nim-waku@a66b8076 - reverse v1 check in container image tag
• 14a1c571 - nim-waku; use deploy-waku-connect docker tag
• e7498861 - requirements: fix apply fix for nim-waku v2 cli flags

 > a nim-waku-connect --become-user=admin -a 'docker ps' 
nim-01.do-ams3.waku.connect | CHANGED | rc=0 >>
CONTAINER ID   NAMES      IMAGE                                     CREATED          STATUS
8f324b48be63   nim-waku   statusteam/nim-waku:deploy-waku-connect   14 minutes ago   Up 14 minutes (healthy)
nim-01.gc-us-central1-a.waku.connect | CHANGED | rc=0 >>
CONTAINER ID   NAMES      IMAGE                                     CREATED          STATUS
269045defa63   nim-waku   statusteam/nim-waku:deploy-waku-connect   14 minutes ago   Up 14 minutes (healthy)
nim-01.ac-cn-hongkong-c.waku.connect | CHANGED | rc=0 >>
CONTAINER ID   NAMES      IMAGE                                     CREATED          STATUS
e2caa90f5a07   nim-waku   statusteam/nim-waku:deploy-waku-connect   14 minutes ago   Up 14 minutes (healthy)

@D4nte does this work fine?

[D4nte]

thanks @jakubgs

For all 3 nim-waku nodes, using the following connection details, I could not connect to websocket port 443 with either libp2p and websocat

• /dns4/nim-01.ac-cn-hongkong-c.waku.connect.statusim.net/tcp/443/wss/p2p/16Uiu2HAm75XUMGev2Ti74G3wUzhyxCtbaDKVWzNwbq3tn5WfzRd4
• /dns4/nim-01.do-ams3.waku.connect.statusim.net/tcp/443/wss/p2p/16Uiu2HAm9VLETt1xBwDAwfKxj2XvAZDw73Bn4HQf11U26JGDxqZD
• /dns4/nim-01.gc-us-central1-a.waku.connect.statusim.net/tcp/443/wss/p2p/16Uiu2HAmMi8xaj9W22a67shGg5wtw1nZDNtfrTPHkgKA5Uhvnvbn

I do not have the details of the go-waku node to test it.

[jakubgs]

Oh yeah, the ports are filtered on two out of 3 hosts. Weird:

 > sudo nmap -Pn -p443 nim-01.do-ams3.waku.connect nim-01.ac-cn-hongkong-c.waku.connect nim-01.gc-us-central1-a.waku.connect              
Nmap scan report for nim-01.do-ams3.waku.connect (206.189.242.0)
Host is up.

PORT    STATE    SERVICE
443/tcp filtered https

Nmap scan report for nim-01.ac-cn-hongkong-c.waku.connect (47.242.185.35)
Host is up (0.28s latency).

PORT    STATE SERVICE
443/tcp open  https

Nmap scan report for nim-01.gc-us-central1-a.waku.connect (35.193.87.35)
Host is up.

PORT    STATE    SERVICE
443/tcp filtered https

Nmap done: 3 IP addresses (3 hosts up) scanned in 3.52 seconds

[jakubgs]

We do have 443 open on the firewall though:

[email protected]:~ % t -n -L SERVICES
Chain SERVICES (2 references)
target     prot opt source               destination         
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:22 /* SSH access */
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:51820 /* WireGuard VPN UDP */
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:30303 /* nim-waku */
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:30303 /* nim-waku */
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:443 /* nim-waku */

[D4nte]

We do have 443 open on the firewall though:

[email protected]:~ % t -n -L SERVICES
Chain SERVICES (2 references)
target     prot opt source               destination         
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:22 /* SSH access */
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:51820 /* WireGuard VPN UDP */
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:30303 /* nim-waku */
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:30303 /* nim-waku */
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:443 /* nim-waku */

I was able to connect to 443 using telnet so if there is no port mismatch then I am I guessing it's a software issue, probably related to waku-org/nwaku#845.

I can try to have a look at the nim-waku logs. Are they easy to find?

Cc @jm-clius

[jakubgs]

It's actually weird. I re-created the container on nim-01.do-ams3.waku.connect and that fixed it:

Nmap scan report for nim-01.do-ams3.waku.connect (206.189.242.0)
Host is up (0.034s latency).

PORT    STATE SERVICE
443/tcp open  https

Nmap scan report for nim-01.ac-cn-hongkong-c.waku.connect (47.242.185.35)
Host is up (0.28s latency).

PORT    STATE SERVICE
443/tcp open  https

Nmap scan report for nim-01.gc-us-central1-a.waku.connect (35.193.87.35)
Host is up.

PORT    STATE    SERVICE
443/tcp filtered https

Nmap done: 3 IP addresses (3 hosts up) scanned in 1.16 seconds

So now I did that on nim-01.gc-us-central1-a.waku.connect and that worked too.

[jakubgs]

The containers have been running for more than a day:

 > a all --become-user=admin -a 'docker ps' 
nim-01.do-ams3.waku.connect | CHANGED | rc=0 >>
CONTAINER ID   NAMES      IMAGE                                     CREATED        STATUS
76b850a47b5b   nim-waku   statusteam/nim-waku:deploy-waku-connect   32 hours ago   Up 32 hours (healthy)
go-01.do-ams3.waku.connect | CHANGED | rc=0 >>
CONTAINER ID   NAMES          IMAGE                            CREATED      STATUS
4f1ce82aa297   go-waku-node   statusteam/go-waku:deploy-test   6 days ago   Up 6 days
nim-01.gc-us-central1-a.waku.connect | CHANGED | rc=0 >>
CONTAINER ID   NAMES      IMAGE                                     CREATED        STATUS
e026bdea8768   nim-waku   statusteam/nim-waku:deploy-waku-connect   30 hours ago   Up 30 hours (healthy)
nim-01.ac-cn-hongkong-c.waku.connect | CHANGED | rc=0 >>
CONTAINER ID   NAMES      IMAGE                                     CREATED      STATUS
8829ed5666a8   nim-waku   statusteam/nim-waku:deploy-waku-connect   2 days ago   Up 2 days (healthy)

But the websocket port is now unresponsive:

 > sudo nmap -Pn -p443 nim-01.do-ams3.waku.connect nim-01.ac-cn-hongkong-c.waku.connect nim-01.gc-us-central1-a.waku.connect
Nmap scan report for nim-01.do-ams3.waku.connect (206.189.242.0)
Host is up.

PORT    STATE    SERVICE
443/tcp filtered https

Nmap scan report for nim-01.ac-cn-hongkong-c.waku.connect (47.242.185.35)
Host is up.

PORT    STATE    SERVICE
443/tcp filtered https

Nmap scan report for nim-01.gc-us-central1-a.waku.connect (35.193.87.35)
Host is up.

PORT    STATE    SERVICE
443/tcp filtered https

So it seems to me like this indeed might be what was described in waku-org/nwaku#845 and status-im/nim-websock#79.

[jakubgs]

@D4nte can this be closed?

[D4nte]

@D4nte can this be closed?

Yes let's close. Can I deploy myself latest master or latest release or do I need to open an issue and ask you?

[jakubgs]

There's a CI job that builds and pushes master by default: https://ci.status.im/job/nim-waku/job/deploy-waku-connect/

[D4nte]

Thanks @jakubgs , one more question: what are the details of the go-waku node?

[jakubgs]

You can check that yourself, both on the host - by looking at docker-compose.yml or inspecting the container - or by looking at the Ansible host group configuration:
https://github.com/status-im/infra-waku-connect/blob/master/ansible/group_vars/go-waku-connect.yml

[D4nte]

You can check that yourself, both on the host - by looking at docker-compose.yml or inspecting the container - or by looking at the Ansible host group configuration: https://github.com/status-im/infra-waku-connect/blob/master/ansible/group_vars/go-waku-connect.yml

Thank you, will check and learn