Connectivity visualization UI

[tomaka]

This issue contains a draft of how to display to the user what smoldot is currently doing in terms of networking.

What I have in mind is: in the popup would be a button next to each chain, and clicking it opens another window that shows a visualization of the connectivity (what this issue is about).
Since the connectivity visualization has nothing clickable and will never have, we could also just display it while the user hovers the button instead of clicking.

Intro: how networking works

Connections

This section contains a small introduction of how the networking in smoldot (and Substrate/Polkadot) works.

A node tries to open connections (typically through WebSocket, but in principle can be anything) to other nodes. These connections can be in two different states: handshaking, or fully established. When we start opening a connection, it is initially in handshaking mode. After the handshake is finished, we know what the PeerId of the remote is, and the connection can be used. We don't know the PeerId of the remote of a connection until after it has finished its handshake.

If there is a problem, the connection shuts down. This can happen (and actually happens most frequently) when the connection is still in handshaking mode. For example, if you try to connect to a port that is closed, the connection shuts down while it is still handshaking.

A Substrate/Polkadot node can also receive incoming connections. It works exactly the same way as when it is the node that initiates the connection: the connection is first handshaking, during which we discover what the PeerId of the remote is, then the connection is fully established.

There is one difference between incoming and outgoing connections: for outgoing connections, while we don't know the actual PeerId before the handshake is finished, we know the expected PeerId. When we establish an outgoing connection, it is always with the intention to connect to a specific PeerId.

There can be multiple connections at the same time towards the same PeerId.

Substreams

Once a connection is fully established (i.e. has finished its handshake), substreams are opened within the connection. It is as if the connection was subdivided into multiple different connections (called substreams) that each do one thing. You have one substream to receive block announces, one substream to send transactions, etc.
Amongst these substreams, there is one substream that can be considered as the "main" one, which is the block announces substream. It is the first substream we open.
Just like connections, substreams are in two different states: handshaking, or fully opened.

Slots

In order for smoldot to know who to try to connect to, it has a certain number of slots and assigns PeerIds to these slots. Right now the number of slots is 4.
When a new chain is initialized, all of its slots are empty. When a slot is empty, smoldot picks a random PeerId amongst the list of PeerIds that it thinks exist, and attributes the slot to this PeerId. While a slot is filled with a PeerId, smoldot tries to connect to this PeerId and open substreams. If we know multiple different multiaddresses for a PeerId, smoldot will try all of them. If it thinks that it is impossible to establish a connection or open the block announces substream, the slot is un-assigned and will later be assigned to a different PeerId.

What I have described in this paragraph is actually "out slots". Smoldot/Substrate/Polkadot also has "in slots", but we can consider them as irrelevant here. They are used in order to prevent too many people from connecting, but knowing which PeerId has an "in slot" isn't very useful for the end user.

The UI

Here is what I have in mind for the connectivity visualization UI:

This box would show a list of PeerIds that are currently assigned to slots, and of PeerIds that were recently assigned to slots, and of PeerIds that we were recently connected to.

If smoldot has connectivity issues, the list of PeerIds will typically change quite rapidly. In order for the UI to be readable, the list should not reorder itself automatically and gaps should not disappear automatically. In other words, PeerIds should visually stay where they are. If a PeerId disappears from the list, it should leave an empty gap that can later be filled with a different PeerId.

Within each PeerId there's a list of multiaddresses of connections to this PeerId: fully established incoming connections, fully established outgoing connections, and handshaking outgoing connections (in which case we use the expected PeerId, as we don't know the actual one yet).

For each multiaddr we show not the state of the connection, but the state of the main substream.
We show a "handshaking" state either if the connection itself is handshaking, or if the block announces substream (i.e. the "main" substream) has not been opened yet.
We show a "fully established" state if the block announces substream is open.
We show a "recently closed" state if the connection itself has been closed recently (which, as I explain above, includes failed to reach the peer for example because the port is closed) or if the block announces substream has been closed recently. In both cases, we show the reason for this closing.

How to implement this

cc paritytech/smoldot#2245

Smoldot will gain support for a JSON-RPC function that sends back networking events.
These networking events would be fed into a state machine that will then report which information should be shown in the UI.

This section is mostly empty because smoldot doesn't support the JSON-RPC function yet anyway. This is something that we will figure later.

[wirednkod]

cc @Goranch3

[Goranch3]

@tomaka @wirednkod thank you for this introduction, a few questions:
1.
"Once a connection is fully established (i.e. has finished its handshake), substreams are opened within the connection."
"Just like connections, substreams are in two different states: handshaking, or fully opened."

Am I getting this right, so connection is fully established, then substreams start their handshakes and once those are finished substreams are working?
2.
Slots, this part gets blurry I re-read it couple of times, I guess the question is what has 4 slots?
So, Node has a PeerId, connections has substreams, what actually has 4 slots and how does it tie to the rest of the story :)
I assume it's the node that has 4 slots but please verify that.

[Goranch3]

The slots are outlined red?

[Goranch3]

and these are PeerIds?

[Goranch3]

And this is a node address?

[Goranch3]

@tomaka can you take a look at the above please, thanks

[tomaka]

For the slots, yes.

PeerIDs are the things that look like 12D3KooWP3BsFY6UaiLjEJ3YbDp6q6SMQgAHB15qKj41DUZQLMqD

Addresses (or multiaddresses) are the things that look like /dns/polkadot-connect-6.parity.io/tcp/443/wss

[Goranch3]

@tomaka @wirednkod thank you for this introduction, a few questions: 1. "Once a connection is fully established (i.e. has finished its handshake), substreams are opened within the connection." "Just like connections, substreams are in two different states: handshaking, or fully opened."

Am I getting this right, so connection is fully established, then substreams start their handshakes and once those are finished substreams are working? 2. Slots, this part gets blurry I re-read it couple of times, I guess the question is what has 4 slots? So, Node has a PeerId, connections has substreams, what actually has 4 slots and how does it tie to the rest of the story :) I assume it's the node that has 4 slots but please verify that.

@tomaka can you please answer these as well

[Goranch3]

Is there a reason why there is a gap between slot 2 and 3 in the mockup you made?

[tomaka]

@tomaka can you please answer these as well

What you said is correct.

When it comes to slots, it is the node that has 4 slots. This means that we have a maximum of 4 peers for which there are substreams open.

I think that there's no need to understand the concept of slots to understand this idea. What is important to highlight is that the number of items in the list will not grow indefinitely. There can temporarily be more items than number of slots, but it should eventually stabilize to 4 items in this list.

Is there a reason why there is a gap between slot 2 and 3 in the mockup you made?

Yes! What I wanted to highlight is that if we disconnect, and that an item is removed, the item should be replaced with an empty space, so that the items below do not immediately "jump up".
In situations where the networking doesn't work (which is the situation where this UI would be the most useful), items might appear and disappear rapidly, and for readability we should IMO avoid having stable items jumping around.

This is a detail, but I thought I would mention it in order to anticipate this.

[Goranch3]

@tomaka
https://user-images.githubusercontent.com/92366349/189342621-60920135-3b16-4f0d-9e9c-c53a94f134f1.mov
This is along the lines you described, I think it has some mayor UX problems, don't know how it looks from your side?
Is there a chance we move this to options, maybe networks tab?

[tomaka]

This indeed has major UX issues, but this doesn't really match what I explain in the OP:

What I have in mind is: in the popup would be a button next to each chain, and clicking it opens another window that shows a visualization of the connectivity (what this issue is about).

The idea was not to display the information directly in the popup, but to have a clickable button.

[wirednkod]

What I have in mind is: in the popup would be a button next to each chain, and clicking it opens another window that shows a visualization of the connectivity (what this issue is about).

Having a popup on a popup, is not the best. Also due to being an extension, once 2nd popup opens (after button click), and user tries to interact with it, the 1st one most likely will close (get destroyed) and 2nd one will (apparently) follow.
IMO best UX solution is to navigate with the click of the button to the Options page and have a tab there containing the information

[Goranch3]

@wirednkod yes exactly. @tomaka how did you imagine this separate widow as another popup or navigating back to options page?

[tomaka]

I think it didn't make this explicit, but I don't think it's worth designing anything at this point.
The best course of action no matter what is to implement it in an ugly way and then make it look better afterwards.
What I'm describing is the first idea. Everything I've described might be completely wrong an unusable and subject to change. Anything you design right now might have to be thrown away later.

[Goranch3]

@tomaka I truly wish we would push this issue towards the solution rather than relativization. Yes, every design might be thrown away that is the nature of it, but we do that based on feedback/data, not personal point of view, I am not married to any UI solution I present but I can pretty reliably tell you if a proposal is within a space that web users find familiar or not.
That said can we agree to present connectivity data inside options page, as it makes most sense?
thanks