From 7c78fa3d5d98290df52b2cda12de9a464800a851 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Tue, 15 Oct 2024 18:21:53 +0200 Subject: [PATCH 01/28] Describe network addressing and translation --- arch-snapshot/arch.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index e962702..b891109 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -589,6 +589,32 @@ block-beta Address2["/market/protocol/mk1/discovery/offers/Get"] ``` +The Net module follows specific GSB address naming conventions to enable cooperation with other modules. Addresses +prefixed with `/net/{NodeId}` are reserved for the Net module, where it listens for incoming messages and forwards +them to the Golem Network. Conversely, addresses starting with `/public/...` are available for yagna modules to expose +public methods that can be called from other Nodes. + +When the Net module receives a local incoming message, it extracts the NodeId from the address prefix and uses it to +forward the message into the Golem Network. On the receiving end, messages coming from the Network are processed, +and the address is checked to extract the NodeId. If the NodeId belongs to the recipient Node, the address is routed to +the appropriate GSB handler registered under the `/public/...` address. + +```mermaid +block-beta + columns 2 + Prefix{{"Prefix"}} + Address{{"Address"}} + + Prefix1["/net/0x467ab03ac10877d0ccff89fac547a4ce8aa0cc5e"] + Address1["/market/protocol/mk1/discovery/offers/Get"] + + arrow1<["Translate"]>(down) + space + + Prefix2["/public"] + Address2["/market/protocol/mk1/discovery/offers/Get"] +``` + ##### Broadcasting Message broadcasting in the Net module is organized around the concept of 'topics,' which can be thought of as From ce55a8e909c6b1e39b2bf401b07aa1085aba1b47 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Wed, 16 Oct 2024 15:13:00 +0200 Subject: [PATCH 02/28] Describe broadcasting --- arch-snapshot/arch.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index b891109..daeb191 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -644,6 +644,8 @@ sequenceDiagram ##### Handling identities + + Each Golem Node can have multiple identities, with one of them (the default identity) used to identify the Node within the network. However, operations on a Golem Node can also be performed in the context of secondary identities. The Net module must be able to handle messages sent to and from any of these identities. For more information on From ca78914c46967c38b1c600e7a493dd330339d7ac Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Wed, 16 Oct 2024 16:26:24 +0200 Subject: [PATCH 03/28] Describe how identities are handled --- arch-snapshot/arch.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index daeb191..bb77f23 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -644,7 +644,16 @@ sequenceDiagram ##### Handling identities +Each Golem Node can have multiple identities, though only one is used to identify the Node within the network. The +Net module must be capable of handling messages sent to and from any of these identities. +In addition to the GSB endpoints bound to the `/net/{NodeId}` prefix, as described in the [Address Translation +chapter](#address-translation), there is another prefix: `/from/{LocalId}/to/{RemoteId}`. This enables messages to +be sent from a specific identity on one Node to a specific identity on a remote Node. + +Another important aspect is that the Net module always checks if the target identity belongs to the local Node. If +it does, the message is routed back to the local GSB instead of being sent over the network. This mechanism allows +GSB calls to be handled uniformly by the calling code, regardless of whether the target is local or remote. Each Golem Node can have multiple identities, with one of them (the default identity) used to identify the Node within the network. However, operations on a Golem Node can also be performed in the context of secondary identities. From 38033e016a55c65011099fea09f80a22b198bd8a Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Wed, 16 Oct 2024 16:57:46 +0200 Subject: [PATCH 04/28] Improve chapter about network identitfication; add link to identity --- arch-snapshot/arch.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index bb77f23..cb125db 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -644,8 +644,11 @@ sequenceDiagram ##### Handling identities -Each Golem Node can have multiple identities, though only one is used to identify the Node within the network. The -Net module must be capable of handling messages sent to and from any of these identities. +Each Golem Node can have multiple identities, with one of them (the default identity) used to identify the Node +within the network. However, operations on a Golem Node can also be performed in the context of secondary identities. +The Net module must be able to handle messages sent to and from any of these identities. For more information on +identification, refer to the chapter about the [identity module](#identity). This section focuses solely on the Net +module interface. In addition to the GSB endpoints bound to the `/net/{NodeId}` prefix, as described in the [Address Translation chapter](#address-translation), there is another prefix: `/from/{LocalId}/to/{RemoteId}`. This enables messages to From b6a03fd73c5be8e46d7de33aa01ddd3a99999ad5 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Wed, 16 Oct 2024 19:33:17 +0200 Subject: [PATCH 05/28] Introduction to hybrid net --- arch-snapshot/arch.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index cb125db..8ffc22c 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -700,7 +700,20 @@ For messages sent from non-default identities, the prefixes are: - `/udp/from/{LocalId}/to/{RemoteId}` - `/transfer/from/{LocalId}/to/{RemoteId}` -#### Hybrid net +#### Hybrid Net + +Hybrid Net was developed as an intermediate step towards decentralization, enabling peer-to-peer (P2P) communication +between Golem Nodes. However, since most of the network operates behind NATs, P2P cannot be the sole communication +method. To address this, the Net implementation supports communication forwarding through specialized Node known +as relay. + +An additional advantage of relay server is it's ability to expedite Node discovery. In a pure P2P network, Node +discovery can be slow, as no single Node has a complete view of the network, requiring multiple hops to find new Nodes. +Relay server can also facilitate P2P communication between Nodes when direct connections are not possible. + + + + - Identification - Relay - Discovering Nodes From 228b5bf17ce3e2ca51f31375c45854159cc059c9 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Thu, 17 Oct 2024 13:24:37 +0200 Subject: [PATCH 06/28] Describe relay server --- arch-snapshot/arch.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index 8ffc22c..5bd28fa 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -711,7 +711,31 @@ An additional advantage of relay server is it's ability to expedite Node discove discovery can be slow, as no single Node has a complete view of the network, requiring multiple hops to find new Nodes. Relay server can also facilitate P2P communication between Nodes when direct connections are not possible. +##### Relay server +The Relay server is a core component of the networking layer in the Golem Network. All newly connected Nodes +register with the Relay server, providing the necessary information for discovery and connection. The Relay server: +- Maintains a list of Nodes present in the network. +- Stores each Node's identity, public key, and IP address. +- Assists in establishing peer-to-peer (p2p) connections when possible. +- Routes traffic between Nodes if a p2p connection cannot be established. +- Offers functions for: + - Discovering Nodes and querying their information. + - Retrieving a Node's neighborhood. + +Communication with the Relay server is handled through a custom protocol built on top of UDP, defined using Protocol +Buffers (protobuf). UDP was chosen for its lightweight nature, as it does not require maintaining open connections, +which would consume more system resources compared to TCP. This makes it possible to handle a large number of Nodes +concurrently, ensuring decent scalability. + +###### Connecting to relay server + + +##### Establishing connections + +##### Network Traffic + +##### Virtual TCP - Identification From 081dc5510a85b013de4562638a154f48252ba229 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Thu, 17 Oct 2024 14:20:33 +0200 Subject: [PATCH 07/28] Diagram: connecting to relay server --- arch-snapshot/arch.md | 40 ++++++++++++++++++++++++++++++---------- 1 file changed, 30 insertions(+), 10 deletions(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index 5bd28fa..ef04430 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -719,6 +719,7 @@ register with the Relay server, providing the necessary information for discover - Stores each Node's identity, public key, and IP address. - Assists in establishing peer-to-peer (p2p) connections when possible. - Routes traffic between Nodes if a p2p connection cannot be established. +- Checks if connecting Nodes have public IP port exposed - Offers functions for: - Discovering Nodes and querying their information. - Retrieving a Node's neighborhood. @@ -730,22 +731,41 @@ concurrently, ensuring decent scalability. ###### Connecting to relay server +```mermaid +sequenceDiagram + participant GolemNode + participant RelayServer + + GolemNode->>RelayServer: Session request + RelayServer->>GolemNode: Challenge + GolemNode->>GolemNode: Solve challenge + GolemNode->>GolemNode: Sign solution with Node's identities + GolemNode->>RelayServer: Challenge response + RelayServer->>RelayServer: Verify solution + RelayServer->>RelayServer: Recover identities from signatures + RelayServer->>GolemNode: Session established response + GolemNode->>RelayServer: Register + activate RelayServer + RelayServer-->>GolemNode: Ping from different UDP port (Check if IP address is public) + GolemNode-->>RelayServer: Ping response + RelayServer->>GolemNode: Register response (discovered public address) + deactivate RelayServer + + Note right of GolemNode: Use session to discover Nodes + + loop In regular intervals to keep session alive + GolemNode->>RelayServer: Ping + RelayServer->>GolemNode: Pong + end +``` -##### Establishing connections +##### Establishing connections between Nodes ##### Network Traffic ##### Virtual TCP - -- Identification -- Relay -- Discovering Nodes -- P2P communication -- Relayed communication -- Cryptography - - Node identity verification (challenges) - - Communication encryption +##### Node identification #### Central net From 43a6ff829faa3364cce26bb05774fcc47787e7f9 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Thu, 17 Oct 2024 17:33:03 +0200 Subject: [PATCH 08/28] Describe Challenge and Public IP check --- arch-snapshot/arch.md | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index ef04430..b6bf712 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -731,6 +731,36 @@ concurrently, ensuring decent scalability. ###### Connecting to relay server +The Hybrid Net protocol introduces the concept of a Session, which operates on top of the UDP protocol. All requests +to the Relay server and network traffic routing occur within the context of a Session. To establish a Session with +the Relay server, a Node must undergo a handshake process that serves several purposes: +1. Verifies that the Node presenting its NodeId possesses the private key corresponding to that NodeId. +2. Collects and verifies any secondary identities associated with the Node. +3. Gathers additional Node information, such as supported symmetric encryption algorithms. +4. Determines if the Node is behind a NAT or if it has a public IP address and exposed port. The public IP acquired in + this step may later be used by other Nodes to establish a peer-to-peer Session. + +**Challenge** + +Identity verification is performed via a challenge mechanism, where the Relay server sends random bytes to the Node, +which must compute a hash with a specified number of leading zeros. The difficulty level, set by the Relay server, +determines the number of leading zeros required. This computationally expensive task protects the Relay server from +DDoS attacks by forcing the Node to complete a certain amount of work before establishing a Session. + +The challenge is cryptographically signed using the private key of each identity associated with the Node. The Relay +server can then recover the Node's identities and public keys from these signatures, verifying that the Node +possesses the corresponding private keys. The recovered public keys are later made available to other Nodes that +request information about the Node. + +**Public IP check** + +When a Node initiates a Session, an additional mechanism is required to determine whether the IP address from which +the packets are received is behind a NAT. This is achieved by sending Ping packets (network protocol ping, not to be +confused with the Linux command) from a different UDP port than the one used for receiving incoming Sessions. This +ensures that any network devices between the Node and the Relay server won't recognize these Ping packets as part of +the same communication stream. If the ports are not publicly exposed, the Ping packets will be dropped, confirming +that the Node is behind a NAT. + ```mermaid sequenceDiagram participant GolemNode @@ -757,14 +787,24 @@ sequenceDiagram GolemNode->>RelayServer: Ping RelayServer->>GolemNode: Pong end + + opt Close Session + GolemNode->>RelayServer: Disconnect + end ``` +Important note: The Relay server does not possess private keys, and its identity is not verified in the current +implementation. This marks a significant distinction compared to the process of establishing peer-to-peer Sessions +with regular Nodes. + ##### Establishing connections between Nodes ##### Network Traffic ##### Virtual TCP +##### Broadcasting + ##### Node identification #### Central net From 65706d7dcbe5f24d8674061691d38253072951d7 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Thu, 17 Oct 2024 17:38:09 +0200 Subject: [PATCH 09/28] IP check: add alternative: timeout when packet is drpped --- arch-snapshot/arch.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index b6bf712..91ee465 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -774,10 +774,15 @@ sequenceDiagram RelayServer->>RelayServer: Verify solution RelayServer->>RelayServer: Recover identities from signatures RelayServer->>GolemNode: Session established response + GolemNode->>RelayServer: Register activate RelayServer RelayServer-->>GolemNode: Ping from different UDP port (Check if IP address is public) - GolemNode-->>RelayServer: Ping response + alt + GolemNode-->>RelayServer: Ping response + else Timeout + GolemNode--xRelayServer: Packet dropped due to NAT + end RelayServer->>GolemNode: Register response (discovered public address) deactivate RelayServer From fa03be5852da2e7501f0a05b77bfbba8f1185ea1 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Thu, 17 Oct 2024 19:54:11 +0200 Subject: [PATCH 10/28] Describe p2p connections between Nodes --- arch-snapshot/arch.md | 50 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 48 insertions(+), 2 deletions(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index 91ee465..143ed06 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -763,8 +763,8 @@ that the Node is behind a NAT. ```mermaid sequenceDiagram - participant GolemNode - participant RelayServer + participant GolemNode as Golem Node 1 + participant RelayServer as Relay Server GolemNode->>RelayServer: Session request RelayServer->>GolemNode: Challenge @@ -804,6 +804,52 @@ with regular Nodes. ##### Establishing connections between Nodes +Currently, a peer-to-peer (p2p) session can be established in two scenarios. In the first, if the target Node has a +public IP, the initiating Node can directly connect to it. In the second scenario, where the initiating Node has a +public IP but the target Node is behind a NAT, the connection is facilitated by the Relay server. The initiating +Node first sends a Reverse Connection message to the Relay server, which forwards it to the target Node. The target +Node then attempts to establish a direct connection with the initiating Node. Whether the target Node has a public +IP can be determined based on information returned by the Relay server. + +Since the current Net implementation does not support NAT punching, if both Nodes are behind NAT, communication must +be routed through the Relay server. + +```mermaid +--- +title: Options to establish P2P connection +--- +sequenceDiagram + participant GolemNode1 as Golem Node 1 + participant RelayServer as Relay Server + participant GolemNode2 as Golem Node 2 + + GolemNode1-->RelayServer: Established Session + GolemNode2-->RelayServer: Established Session + + GolemNode1->>RelayServer: Get Node 2's information + RelayServer->>GolemNode1: Node 2's information + + alt Golem Node 2 has public IP + GolemNode1->>GolemNode2: Establish P2P connection + else Golem Node 1 has public IP + GolemNode1->>RelayServer: Reverse Connection message + RelayServer->>GolemNode2: Reverse Connection message + + GolemNode2->>RelayServer: Get Node 1's information + RelayServer->>GolemNode2: Node 1's information + + GolemNode2->>GolemNode1: Establish P2P connection + else Golem Node 1 and 2 are behind NAT + par Communication routed through relay + GolemNode1->>RelayServer: Forward packet + RelayServer->>GolemNode2: Forward packet + GolemNode2->>RelayServer: Forward packet + RelayServer->>GolemNode1: Forward packet + end + end +``` + + ##### Network Traffic ##### Virtual TCP From a2514f7832a90d2d1b973088673056cbdb6688de Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Thu, 17 Oct 2024 20:11:29 +0200 Subject: [PATCH 11/28] Describe p2p handshake --- arch-snapshot/arch.md | 48 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 47 insertions(+), 1 deletion(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index 143ed06..efe35a7 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -816,7 +816,7 @@ be routed through the Relay server. ```mermaid --- -title: Options to establish P2P connection +title: Scenarios of communication between Nodes --- sequenceDiagram participant GolemNode1 as Golem Node 1 @@ -849,6 +849,52 @@ sequenceDiagram end ``` +**Peer-to-peer connection handshake** + +Establishing a peer-to-peer (p2p) connection between Nodes is similar to the Relay server handshake but with a few +key differences: +- Both Nodes must solve a challenge and prove their identities to each other. +- There is no registration step or public IP check, as the public IP of the Nodes is already known. +- The initiating Node sends a "Resume Forwarding" control message to the target Node, informing it that it can begin + sending packets. This step ensures that packets arriving too early are not dropped. + +```mermaid +--- +title: Protocol for establishing peer-to-peer Session +--- +sequenceDiagram + participant GolemNode1 as Golem Node 1 + participant GolemNode2 as Golem Node 2 + + GolemNode1->>GolemNode2: Session request (+challenge) + GolemNode2->>GolemNode1: Challenge + + GolemNode1->>GolemNode1: Solve challenge + GolemNode2->>GolemNode2: Solve challenge + GolemNode1->>GolemNode1: Sign solution with Node's identities + GolemNode2->>GolemNode2: Sign solution with Node's identities + + GolemNode1->>GolemNode2: Challenge response + GolemNode2->>GolemNode2: Verify solution + GolemNode2->>GolemNode2: Recover identities from signatures + GolemNode2->>GolemNode1: Challenge response + + GolemNode1->>GolemNode1: Verify solution + GolemNode1->>GolemNode1: Recover identities from signatures + Note over GolemNode1: Node can start sending packets from this moment + + GolemNode1->>GolemNode2: Resume Forwarding control message + Note over GolemNode2: Node can start sending packets from this moment + + loop In regular intervals to keep session alive + GolemNode1->>GolemNode2: Ping + GolemNode2->>GolemNode1: Pong + end + + opt Close Session + GolemNode1->>GolemNode2: Disconnect + end +``` ##### Network Traffic From ac872c84c54df10fdced6029048e6a6994425269 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Fri, 18 Oct 2024 12:34:39 +0200 Subject: [PATCH 12/28] Describe Forwarding network traffic and virtual TCP --- arch-snapshot/arch.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index efe35a7..a3948ff 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -896,10 +896,20 @@ sequenceDiagram end ``` -##### Network Traffic +##### Forwarding Network Traffic + +The low-level abstraction provides a single message type for sending data: the `Forward` packet. This packet can be +used to send arbitrary content between Nodes, either directly or through the Relay server. Like UDP, the Forward +packet does not offer delivery guarantees. It is the responsibility of higher-level layers to ensure the correct and +reliable delivery of data in case it is necessary. ##### Virtual TCP +To enable reliable data delivery, the hybrid Net utilizes an embedded TCP stack implementation. This stack takes an +input data stream and generates IP packets, which are then sent using Forward packets. The receiving Node employs +the TCP stack to decode incoming packets back into a message stream. These messages are subsequently dispatched as +GSB messages and passed to the appropriate modules, as detailed in the [Address Translation chapter](#address-translation). + ##### Broadcasting ##### Node identification From 70c896e6baba93e1d38c841f4ba79bdf848345bc Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Fri, 18 Oct 2024 12:44:08 +0200 Subject: [PATCH 13/28] Describe how different channels are implemented in Hybrid Net --- arch-snapshot/arch.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index a3948ff..3665c20 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -910,6 +910,17 @@ input data stream and generates IP packets, which are then sent using Forward pa the TCP stack to decode incoming packets back into a message stream. These messages are subsequently dispatched as GSB messages and passed to the appropriate modules, as detailed in the [Address Translation chapter](#address-translation). +##### Reliable, unreliable and transfers channels in Hybrid Net + +Different channels can be utilized to send messages, as explained in the [reliable, unreliable, and transfers +channels chapter](#reliable-unreliable-and-transfers-channels). In the Hybrid Net, reliable and transfer channels +are distinguished by using separate TCP connections. This separation ensures that independent sender buffers are +maintained, preventing messages in one channel from being blocked by messages in the other. + +The sender can also opt to use the unreliable channel, where GSB messages are sent directly as `Forward` packets +without message fragmentation. A key implication of this is that large GSB messages could exceed the Maximum +Transmission Unit (MTU) and may be dropped by network devices along the packet's route. + ##### Broadcasting ##### Node identification From d5b42f783134695122969dd5243b48bb694d8365 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Fri, 18 Oct 2024 14:08:55 +0200 Subject: [PATCH 14/28] Describe broadcasts to neighborhood --- arch-snapshot/arch.md | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index 3665c20..7eb8fb8 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -921,9 +921,26 @@ The sender can also opt to use the unreliable channel, where GSB messages are se without message fragmentation. A key implication of this is that large GSB messages could exceed the Maximum Transmission Unit (MTU) and may be dropped by network devices along the packet's route. -##### Broadcasting +##### Broadcasting and neighborhood + +Hybrid Net implements local broadcasting to the nearest neighborhood of each Node. To query its neighbors, a Node +can send a `Neighborhood` request to the Relay server. The Relay server then responds with a list of Nodes that are +closest to the querying Node, based on a predefined metric. + +After receiving the list of neighbors, the Node attempts to establish connections with them, as described in the +chapter on [communication](#establishing-connections-between-nodes). The neighborhood algorithm does not +differentiate between Nodes capable of establishing peer-to-peer connections and those that require relayed +communication. Unlike IP-level broadcasts, Hybrid Net uses reliable channels for message transmission. + +**Neighborhood - distance function** + +To prevent clustering of Nodes and accidental splits in the network, where subsets of Nodes become unreachable, a proper +neighborhood function must be utilized. This function is defined by the network module, and the market relies on the +broadcast function, leaving it with no alternative in this regard. -##### Node identification +Optimal guarantees can be achieved when two neighboring Nodes have distinctly different neighborhoods, minimizing their +number of common neighbors. Currently, in the Hybrid Net, neighborhood is determined based on the reversed Hamming +distance between Node IDs. #### Central net From d9ad6f95f7230d93e57e01761b4bc071452b2334 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Mon, 21 Oct 2024 13:18:54 +0200 Subject: [PATCH 15/28] Apply trivial review comments --- arch-snapshot/arch.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index 7eb8fb8..360af6a 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -704,8 +704,8 @@ For messages sent from non-default identities, the prefixes are: Hybrid Net was developed as an intermediate step towards decentralization, enabling peer-to-peer (P2P) communication between Golem Nodes. However, since most of the network operates behind NATs, P2P cannot be the sole communication -method. To address this, the Net implementation supports communication forwarding through specialized Node known -as relay. +method. To address this, the Net implementation supports communication forwarding through specialized component known +as Relay. An additional advantage of relay server is it's ability to expedite Node discovery. In a pure P2P network, Node discovery can be slow, as no single Node has a complete view of the network, requiring multiple hops to find new Nodes. @@ -721,7 +721,7 @@ register with the Relay server, providing the necessary information for discover - Routes traffic between Nodes if a p2p connection cannot be established. - Checks if connecting Nodes have public IP port exposed - Offers functions for: - - Discovering Nodes and querying their information. + - Querying Node's information. - Retrieving a Node's neighborhood. Communication with the Relay server is handled through a custom protocol built on top of UDP, defined using Protocol @@ -745,10 +745,10 @@ the Relay server, a Node must undergo a handshake process that serves several pu Identity verification is performed via a challenge mechanism, where the Relay server sends random bytes to the Node, which must compute a hash with a specified number of leading zeros. The difficulty level, set by the Relay server, determines the number of leading zeros required. This computationally expensive task protects the Relay server from -DDoS attacks by forcing the Node to complete a certain amount of work before establishing a Session. +DDoS attacks by forcing the Node to complete a certain amount of work before establishing a Session. -The challenge is cryptographically signed using the private key of each identity associated with the Node. The Relay -server can then recover the Node's identities and public keys from these signatures, verifying that the Node +The challenge response is cryptographically signed using the private key of each identity associated with the Node. +The Relay server can then recover the Node's identities and public keys from these signatures, verifying that the Node possesses the corresponding private keys. The recovered public keys are later made available to other Nodes that request information about the Node. @@ -756,7 +756,7 @@ request information about the Node. When a Node initiates a Session, an additional mechanism is required to determine whether the IP address from which the packets are received is behind a NAT. This is achieved by sending Ping packets (network protocol ping, not to be -confused with the Linux command) from a different UDP port than the one used for receiving incoming Sessions. This +confused with the ICMP packet) from a different UDP port than the one used for receiving incoming Sessions. This ensures that any network devices between the Node and the Relay server won't recognize these Ping packets as part of the same communication stream. If the ports are not publicly exposed, the Ping packets will be dropped, confirming that the Node is behind a NAT. @@ -809,9 +809,9 @@ public IP, the initiating Node can directly connect to it. In the second scenari public IP but the target Node is behind a NAT, the connection is facilitated by the Relay server. The initiating Node first sends a Reverse Connection message to the Relay server, which forwards it to the target Node. The target Node then attempts to establish a direct connection with the initiating Node. Whether the target Node has a public -IP can be determined based on information returned by the Relay server. +IP can be determined based on information returned by the Relay server. -Since the current Net implementation does not support NAT punching, if both Nodes are behind NAT, communication must +Since the current Net implementation does not support NAT hole punching, if both Nodes are behind NAT, communication must be routed through the Relay server. ```mermaid From a7cbb5d150d209d8b2818164d1a0616c1f2ace99 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Mon, 21 Oct 2024 18:16:20 +0200 Subject: [PATCH 16/28] Raw description of neighborhood --- arch-snapshot/arch.md | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index 360af6a..21c7ed6 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -923,6 +923,24 @@ Transmission Unit (MTU) and may be dropped by network devices along the packet's ##### Broadcasting and neighborhood +**What is neighborhood?** + +Before describing broadcasting the concept of neighborhood must be introduced. Neighborhood is subset of Nodes in the +network that are closest to a given Node in terms of some abstract metric. Each Node has it's own neighborhood. +This metric doesn't need to have any relation to any real world closeness of Nodes. For example Nodes on other part of +the globe can be neighbors, whereas 2 Nodes in the same physical network could be too far to be in neighborhood. + +In peer-to-peer networks concept of Nodes distance is often used for faster Node discovery. Since current Golem network +layer implementation uses Relay server for finding Nodes, neighborhood doesn't serve this purpose. Although it can be +easily imaginable to have kademlia implementation as a fallback in case of Relay downtime, currently the only reason for +having this concept is broadcasting. + +Broadcasting can be used by various algorithms to propagate certain information through the network. The most important +usage of this functionality is [Offers propagation](#offer-propagation) algorithm. Implementing algorithms is +responsibility of other modules, net module provides only set of operations that can be used as building blocks. + +**Broadcasting** + Hybrid Net implements local broadcasting to the nearest neighborhood of each Node. To query its neighbors, a Node can send a `Neighborhood` request to the Relay server. The Relay server then responds with a list of Nodes that are closest to the querying Node, based on a predefined metric. @@ -932,7 +950,7 @@ chapter on [communication](#establishing-connections-between-nodes). The neighbo differentiate between Nodes capable of establishing peer-to-peer connections and those that require relayed communication. Unlike IP-level broadcasts, Hybrid Net uses reliable channels for message transmission. -**Neighborhood - distance function** +**Neighborhood - distance function** To prevent clustering of Nodes and accidental splits in the network, where subsets of Nodes become unreachable, a proper neighborhood function must be utilized. This function is defined by the network module, and the market relies on the From 509989fc6b9ffddff9abf3d1c046247252b9cca2 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Tue, 22 Oct 2024 10:51:43 +0200 Subject: [PATCH 17/28] Improve neighborhood chapter --- arch-snapshot/arch.md | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index 21c7ed6..d54495a 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -923,21 +923,23 @@ Transmission Unit (MTU) and may be dropped by network devices along the packet's ##### Broadcasting and neighborhood -**What is neighborhood?** - -Before describing broadcasting the concept of neighborhood must be introduced. Neighborhood is subset of Nodes in the -network that are closest to a given Node in terms of some abstract metric. Each Node has it's own neighborhood. -This metric doesn't need to have any relation to any real world closeness of Nodes. For example Nodes on other part of -the globe can be neighbors, whereas 2 Nodes in the same physical network could be too far to be in neighborhood. - -In peer-to-peer networks concept of Nodes distance is often used for faster Node discovery. Since current Golem network -layer implementation uses Relay server for finding Nodes, neighborhood doesn't serve this purpose. Although it can be -easily imaginable to have kademlia implementation as a fallback in case of Relay downtime, currently the only reason for -having this concept is broadcasting. - -Broadcasting can be used by various algorithms to propagate certain information through the network. The most important -usage of this functionality is [Offers propagation](#offer-propagation) algorithm. Implementing algorithms is -responsibility of other modules, net module provides only set of operations that can be used as building blocks. +**What is a Neighborhood?** + +Before diving into broadcasting, it's essential to understand the concept of a neighborhood. In a network, a +neighborhood is a subset of Nodes that are considered closest to a given Node based on an abstract metric. Each Node +has its own neighborhood. This metric doesn't necessarily reflect the real-world proximity of Nodes. For instance, +two Nodes on opposite sides of the globe could be neighbors, while two Nodes within the same physical network may be +too distant in terms of this metric to be in the same neighborhood. + +In peer-to-peer networks, the concept of Node distance is often used for efficient Node discovery. However, since +the current Golem network layer relies on the Relay server for Node discovery, neighborhoods don't serve that +purpose. Although it’s conceivable that a fallback, such as a Kademlia-like implementation, could be used in the +event of a Relay server downtime, for now, the primary purpose of the neighborhood concept is broadcasting. + +Broadcasting is a mechanism used by various algorithms to propagate information throughout the network. Its most +significant application is in [the Offer Propagation algorithm](#offer-propagation). While the implementation of +specific algorithms is handled by other modules, the network module provides the necessary operations as building +blocks for these processes. **Broadcasting** From 2bd48ef3cbe426e6ab9a15985e242020c9558552 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Wed, 23 Oct 2024 13:01:46 +0200 Subject: [PATCH 18/28] Apply trivial review fixes --- arch-snapshot/arch.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index d54495a..7de1701 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -703,9 +703,9 @@ For messages sent from non-default identities, the prefixes are: #### Hybrid Net Hybrid Net was developed as an intermediate step towards decentralization, enabling peer-to-peer (P2P) communication -between Golem Nodes. However, since most of the network operates behind NATs, P2P cannot be the sole communication -method. To address this, the Net implementation supports communication forwarding through specialized component known -as Relay. +between some of Golem Nodes. However, since most of the network operates behind NATs, P2P cannot be the sole +communication method. To address this, the Net implementation supports communication forwarding through specialized +component known as Relay. An additional advantage of relay server is it's ability to expedite Node discovery. In a pure P2P network, Node discovery can be slow, as no single Node has a complete view of the network, requiring multiple hops to find new Nodes. From 82f07cf4ecede89e4069f862f57d5910d2bb0c87 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Wed, 23 Oct 2024 20:19:37 +0200 Subject: [PATCH 19/28] Describe Node idenitification; Start describing encryption --- arch-snapshot/arch.md | 40 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index 7de1701..4b9c020 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -711,12 +711,23 @@ An additional advantage of relay server is it's ability to expedite Node discove discovery can be slow, as no single Node has a complete view of the network, requiring multiple hops to find new Nodes. Relay server can also facilitate P2P communication between Nodes when direct connections are not possible. +##### Nodes identification + +A Golem Node is identified by its NodeId, which is derived from its public key. This NodeId allows the Node to be +located within the network, and the public key is used to verify the Node’s identity. This ensures that only one +Node can claim a specific ID within the network. + +Additionally, each Node may have multiple secondary identities, each associated with its own public/private key pair. +The Net module must be capable of locating a Golem Node based on these secondary identities as well as the primary +identity. + ##### Relay server The Relay server is a core component of the networking layer in the Golem Network. All newly connected Nodes register with the Relay server, providing the necessary information for discovery and connection. The Relay server: - Maintains a list of Nodes present in the network. -- Stores each Node's identity, public key, and IP address. +- Stores each Node's public key, [identity](#identity) derived from the public key, and associated IP address. +- Stores information about which secondary identities are associated with specific Nodes. - Assists in establishing peer-to-peer (p2p) connections when possible. - Routes traffic between Nodes if a p2p connection cannot be established. - Checks if connecting Nodes have public IP port exposed @@ -962,6 +973,33 @@ Optimal guarantees can be achieved when two neighboring Nodes have distinctly di number of common neighbors. Currently, in the Hybrid Net, neighborhood is determined based on the reversed Hamming distance between Node IDs. +##### Encryption + +The currently released version of Hybrid Net does not support encryption, but this feature is in progress. This +chapter is important to help the reader understand the relationship between identities and how different keys will +be used once encryption is implemented. + +The [Nodes identification chapter](#nodes-identification) explains how identities are used within the Network. +However, the public key pairs associated with these identities are only used to verify whether a Node accurately +claims its identity. For encryption purposes, symmetric keys are employed, which are not related to the identity keys. + +**Symmetric encryption algorithm choice** + +The encryption algorithm chosen must meet the following criteria: +1. Independent Message Encryption: The algorithm must not assume an order for messages; each packet must be + encrypted independently. Since the entire network traffic between Nodes should be encrypted, and part of the + communication occurs over an unreliable channel, the encryption algorithm must be capable of handling individual + UDP packets. Some packets may be lost, while others could arrive out of order. +2. No Key Exchange Required: There should be no need for explicit key or information exchange between Nodes. In cases + where communication is relayed, there is no session-establishing phase between parties sending relayed packets, + so key exchanges aren't feasible. + +For these reasons, the AES-GCM-SIV variant of the AES algorithm was chosen. + +**Symmetric keys derivation** + + + #### Central net ### Market interactions From 0e92dff815f9a5ec0f50ff98ba7464ce1376cb92 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Thu, 24 Oct 2024 14:06:23 +0200 Subject: [PATCH 20/28] Apply distinctions between connection and session; Broadcastign terminology --- arch-snapshot/arch.md | 56 +++++++++++++++++++++++-------------------- 1 file changed, 30 insertions(+), 26 deletions(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index 4b9c020..dc6bf4d 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -711,6 +711,9 @@ An additional advantage of relay server is it's ability to expedite Node discove discovery can be slow, as no single Node has a complete view of the network, requiring multiple hops to find new Nodes. Relay server can also facilitate P2P communication between Nodes when direct connections are not possible. +Although HybridNet remains centralized for node discovery and relaying communication between Nodes behind NATs, +broadcasting has already been designed in a decentralized manner as a step toward full network decentralization. + ##### Nodes identification A Golem Node is identified by its NodeId, which is derived from its public key. This NodeId allows the Node to be @@ -724,12 +727,12 @@ identity. ##### Relay server The Relay server is a core component of the networking layer in the Golem Network. All newly connected Nodes -register with the Relay server, providing the necessary information for discovery and connection. The Relay server: +register with the Relay server, providing the necessary information for discovery and communication. The Relay server: - Maintains a list of Nodes present in the network. - Stores each Node's public key, [identity](#identity) derived from the public key, and associated IP address. - Stores information about which secondary identities are associated with specific Nodes. -- Assists in establishing peer-to-peer (p2p) connections when possible. -- Routes traffic between Nodes if a p2p connection cannot be established. +- Assists in establishing peer-to-peer (p2p) communication when possible. +- Routes traffic between Nodes if a p2p communication cannot be established. - Checks if connecting Nodes have public IP port exposed - Offers functions for: - Querying Node's information. @@ -740,11 +743,12 @@ Buffers (protobuf). UDP was chosen for its lightweight nature, as it does not re which would consume more system resources compared to TCP. This makes it possible to handle a large number of Nodes concurrently, ensuring decent scalability. -###### Connecting to relay server +###### Connecting to Relay server The Hybrid Net protocol introduces the concept of a Session, which operates on top of the UDP protocol. All requests -to the Relay server and network traffic routing occur within the context of a Session. To establish a Session with -the Relay server, a Node must undergo a handshake process that serves several purposes: +to the Relay server and network traffic routing occur within the context of a Session, with only one Session allowed +at a time. To establish a Session with the Relay server, a Node must undergo a handshake process that serves several +purposes: 1. Verifies that the Node presenting its NodeId possesses the private key corresponding to that NodeId. 2. Collects and verifies any secondary identities associated with the Node. 3. Gathers additional Node information, such as supported symmetric encryption algorithms. @@ -813,17 +817,17 @@ Important note: The Relay server does not possess private keys, and its identity implementation. This marks a significant distinction compared to the process of establishing peer-to-peer Sessions with regular Nodes. -##### Establishing connections between Nodes +##### Establishing Sessions between Nodes -Currently, a peer-to-peer (p2p) session can be established in two scenarios. In the first, if the target Node has a +Currently, a peer-to-peer (p2p) Session can be established in two scenarios. In the first, if the target Node has a public IP, the initiating Node can directly connect to it. In the second scenario, where the initiating Node has a -public IP but the target Node is behind a NAT, the connection is facilitated by the Relay server. The initiating +public IP but the target Node is behind a NAT, the Session is facilitated by the Relay server. The initiating Node first sends a Reverse Connection message to the Relay server, which forwards it to the target Node. The target -Node then attempts to establish a direct connection with the initiating Node. Whether the target Node has a public +Node then attempts to establish a direct Session with the initiating Node. Whether the target Node has a public IP can be determined based on information returned by the Relay server. -Since the current Net implementation does not support NAT hole punching, if both Nodes are behind NAT, communication must -be routed through the Relay server. +Since the current Net implementation does not support NAT hole punching, if both Nodes are behind NAT, communication +must be routed through the Relay server. ```mermaid --- @@ -835,13 +839,13 @@ sequenceDiagram participant GolemNode2 as Golem Node 2 GolemNode1-->RelayServer: Established Session - GolemNode2-->RelayServer: Established Session + GolemNode2-->RelayServer: Established Session GolemNode1->>RelayServer: Get Node 2's information RelayServer->>GolemNode1: Node 2's information alt Golem Node 2 has public IP - GolemNode1->>GolemNode2: Establish P2P connection + GolemNode1->>GolemNode2: Establish P2P Session else Golem Node 1 has public IP GolemNode1->>RelayServer: Reverse Connection message RelayServer->>GolemNode2: Reverse Connection message @@ -849,7 +853,7 @@ sequenceDiagram GolemNode2->>RelayServer: Get Node 1's information RelayServer->>GolemNode2: Node 1's information - GolemNode2->>GolemNode1: Establish P2P connection + GolemNode2->>GolemNode1: Establish P2P Session else Golem Node 1 and 2 are behind NAT par Communication routed through relay GolemNode1->>RelayServer: Forward packet @@ -860,9 +864,9 @@ sequenceDiagram end ``` -**Peer-to-peer connection handshake** +**Peer-to-peer Session handshake** -Establishing a peer-to-peer (p2p) connection between Nodes is similar to the Relay server handshake but with a few +Establishing a peer-to-peer (p2p) Session between Nodes is similar to the Relay server handshake but with a few key differences: - Both Nodes must solve a challenge and prove their identities to each other. - There is no registration step or public IP check, as the public IP of the Nodes is already known. @@ -897,7 +901,7 @@ sequenceDiagram GolemNode1->>GolemNode2: Resume Forwarding control message Note over GolemNode2: Node can start sending packets from this moment - loop In regular intervals to keep session alive + loop In regular intervals to keep Session alive GolemNode1->>GolemNode2: Ping GolemNode2->>GolemNode1: Pong end @@ -947,20 +951,20 @@ the current Golem network layer relies on the Relay server for Node discovery, n purpose. Although it’s conceivable that a fallback, such as a Kademlia-like implementation, could be used in the event of a Relay server downtime, for now, the primary purpose of the neighborhood concept is broadcasting. -Broadcasting is a mechanism used by various algorithms to propagate information throughout the network. Its most -significant application is in [the Offer Propagation algorithm](#offer-propagation). While the implementation of +Sending broadcast message is an operation used by various algorithms to disseminate information throughout the network. +Its most significant application is in [the Offer Propagation algorithm](#offer-propagation). While the implementation of specific algorithms is handled by other modules, the network module provides the necessary operations as building blocks for these processes. **Broadcasting** -Hybrid Net implements local broadcasting to the nearest neighborhood of each Node. To query its neighbors, a Node -can send a `Neighborhood` request to the Relay server. The Relay server then responds with a list of Nodes that are -closest to the querying Node, based on a predefined metric. +Hybrid Net implements local broadcast operation that sends message to the nearest neighborhood of the Node. To query +its neighbors, a Node can send a `Neighborhood` request to the Relay server. The Relay server then responds with a +list of Nodes that are closest to the querying Node, based on a predefined metric. -After receiving the list of neighbors, the Node attempts to establish connections with them, as described in the +After receiving the list of neighbors, the Node attempts to establish Sessions with them, as described in the chapter on [communication](#establishing-connections-between-nodes). The neighborhood algorithm does not -differentiate between Nodes capable of establishing peer-to-peer connections and those that require relayed +differentiate between Nodes capable of establishing peer-to-peer Sessions and those that require relayed communication. Unlike IP-level broadcasts, Hybrid Net uses reliable channels for message transmission. **Neighborhood - distance function** @@ -991,7 +995,7 @@ The encryption algorithm chosen must meet the following criteria: communication occurs over an unreliable channel, the encryption algorithm must be capable of handling individual UDP packets. Some packets may be lost, while others could arrive out of order. 2. No Key Exchange Required: There should be no need for explicit key or information exchange between Nodes. In cases - where communication is relayed, there is no session-establishing phase between parties sending relayed packets, + where communication is relayed, there is no Session-establishing phase between parties sending relayed packets, so key exchanges aren't feasible. For these reasons, the AES-GCM-SIV variant of the AES algorithm was chosen. From 583a93130a0fcb5ec0474e00408a15e0819bd8a6 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Thu, 24 Oct 2024 16:34:40 +0200 Subject: [PATCH 21/28] Describe key dervation and symmetric encryption --- arch-snapshot/arch.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index dc6bf4d..fb5e924 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -930,7 +930,8 @@ GSB messages and passed to the appropriate modules, as detailed in the [Address Different channels can be utilized to send messages, as explained in the [reliable, unreliable, and transfers channels chapter](#reliable-unreliable-and-transfers-channels). In the Hybrid Net, reliable and transfer channels are distinguished by using separate TCP connections. This separation ensures that independent sender buffers are -maintained, preventing messages in one channel from being blocked by messages in the other. +maintained, preventing messages in one channel from being blocked by messages in the other. Each channel type has +its own single TCP connection. This means that independent transfer still can interfere with each other. The sender can also opt to use the unreliable channel, where GSB messages are sent directly as `Forward` packets without message fragmentation. A key implication of this is that large GSB messages could exceed the Maximum @@ -1002,7 +1003,16 @@ For these reasons, the AES-GCM-SIV variant of the AES algorithm was chosen. **Symmetric keys derivation** +Hybrid Net employs Elliptic Curve Diffie-Hellman (ECDH) to derive symmetric encryption keys. Each Node generates a +new private/public key pair, separate from its identity keys, which is specifically used for communication. The +public key is exchanged during the [p2p handshake](#establishing-sessions-between-nodes), and a shared secret is +derived from this key to enable AES encryption. +In cases of relayed communication, where no direct Session is established, the public key is retrieved from the +Relay server. The shared secret is then derived using the Node's private key, enabling encrypted communication +through the Relay. The recipient Node also requests the sender's public key from the Relay server and uses it to +derive the shared secret, allowing it to decrypt the message. No special control packet exchanges are required +between the sender and receiver. #### Central net From 2b4d8bd9899de44798e09b834c250cd70cf5a1e7 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Wed, 30 Oct 2024 11:31:55 +0100 Subject: [PATCH 22/28] Remove triling spaces --- arch-snapshot/arch.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index fb5e924..73d0df9 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -592,12 +592,12 @@ block-beta The Net module follows specific GSB address naming conventions to enable cooperation with other modules. Addresses prefixed with `/net/{NodeId}` are reserved for the Net module, where it listens for incoming messages and forwards them to the Golem Network. Conversely, addresses starting with `/public/...` are available for yagna modules to expose -public methods that can be called from other Nodes. +public methods that can be called from other Nodes. When the Net module receives a local incoming message, it extracts the NodeId from the address prefix and uses it to forward the message into the Golem Network. On the receiving end, messages coming from the Network are processed, and the address is checked to extract the NodeId. If the NodeId belongs to the recipient Node, the address is routed to -the appropriate GSB handler registered under the `/public/...` address. +the appropriate GSB handler registered under the `/public/...` address. ```mermaid block-beta @@ -916,7 +916,7 @@ sequenceDiagram The low-level abstraction provides a single message type for sending data: the `Forward` packet. This packet can be used to send arbitrary content between Nodes, either directly or through the Relay server. Like UDP, the Forward packet does not offer delivery guarantees. It is the responsibility of higher-level layers to ensure the correct and -reliable delivery of data in case it is necessary. +reliable delivery of data in case it is necessary. ##### Virtual TCP @@ -961,12 +961,12 @@ blocks for these processes. Hybrid Net implements local broadcast operation that sends message to the nearest neighborhood of the Node. To query its neighbors, a Node can send a `Neighborhood` request to the Relay server. The Relay server then responds with a -list of Nodes that are closest to the querying Node, based on a predefined metric. +list of Nodes that are closest to the querying Node, based on a predefined metric. After receiving the list of neighbors, the Node attempts to establish Sessions with them, as described in the chapter on [communication](#establishing-connections-between-nodes). The neighborhood algorithm does not differentiate between Nodes capable of establishing peer-to-peer Sessions and those that require relayed -communication. Unlike IP-level broadcasts, Hybrid Net uses reliable channels for message transmission. +communication. Unlike IP-level broadcasts, Hybrid Net uses reliable channels for message transmission. **Neighborhood - distance function** From 49fdd29004d1a1316764b7dde10e7d2a11904a65 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Wed, 30 Oct 2024 13:46:40 +0100 Subject: [PATCH 23/28] Rephrase broadcasting description; add example --- arch-snapshot/arch.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index 73d0df9..5eb9b59 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -952,8 +952,10 @@ the current Golem network layer relies on the Relay server for Node discovery, n purpose. Although it’s conceivable that a fallback, such as a Kademlia-like implementation, could be used in the event of a Relay server downtime, for now, the primary purpose of the neighborhood concept is broadcasting. -Sending broadcast message is an operation used by various algorithms to disseminate information throughout the network. -Its most significant application is in [the Offer Propagation algorithm](#offer-propagation). While the implementation of +A technique used in various broadcasting algorithms ([example](https://docs.libp2p.io/concepts/pubsub/overview/)) +involves disseminating information through the network by having each node send broadcast messages only to its +neighbors, relying on those neighbors to forward the information further. HybridNet uses it as well. Its most +significant application is in [the Offer Propagation algorithm](#offer-propagation). While the implementation of specific algorithms is handled by other modules, the network module provides the necessary operations as building blocks for these processes. @@ -964,7 +966,7 @@ its neighbors, a Node can send a `Neighborhood` request to the Relay server. The list of Nodes that are closest to the querying Node, based on a predefined metric. After receiving the list of neighbors, the Node attempts to establish Sessions with them, as described in the -chapter on [communication](#establishing-connections-between-nodes). The neighborhood algorithm does not +chapter on [communication](#establishing-sessions-between-nodes). The neighborhood algorithm does not differentiate between Nodes capable of establishing peer-to-peer Sessions and those that require relayed communication. Unlike IP-level broadcasts, Hybrid Net uses reliable channels for message transmission. From 2108efebf1beb8591fb4ea57115a72a54d5b6c6b Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Wed, 30 Oct 2024 14:21:13 +0100 Subject: [PATCH 24/28] Explicite mention that transport types use underying session --- arch-snapshot/arch.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index 5eb9b59..4b90d3d 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -925,18 +925,22 @@ input data stream and generates IP packets, which are then sent using Forward pa the TCP stack to decode incoming packets back into a message stream. These messages are subsequently dispatched as GSB messages and passed to the appropriate modules, as detailed in the [Address Translation chapter](#address-translation). -##### Reliable, unreliable and transfers channels in Hybrid Net +##### Reliable, unreliable and transfers transport types in Hybrid Net -Different channels can be utilized to send messages, as explained in the [reliable, unreliable, and transfers -channels chapter](#reliable-unreliable-and-transfers-channels). In the Hybrid Net, reliable and transfer channels -are distinguished by using separate TCP connections. This separation ensures that independent sender buffers are -maintained, preventing messages in one channel from being blocked by messages in the other. Each channel type has -its own single TCP connection. This means that independent transfer still can interfere with each other. +Different transport types can be utilized to send messages, as explained in the [reliable, unreliable, and transfers +channels chapter](#reliable-unreliable-and-transfers-channels). In the Hybrid Net, reliable and transfer transport +types are distinguished by using separate TCP connections. This separation ensures that independent sender buffers +are maintained, preventing messages in one channel from being blocked by messages in the other. Each transport type +has its own single TCP connection. This means that independent transfer still can interfere with each other. -The sender can also opt to use the unreliable channel, where GSB messages are sent directly as `Forward` packets +The sender can also opt to use the unreliable transport, where GSB messages are sent directly as `Forward` packets without message fragmentation. A key implication of this is that large GSB messages could exceed the Maximum Transmission Unit (MTU) and may be dropped by network devices along the packet's route. +Transport channels are built on top of the underlying [sessions](#establishing-sessions-between-nodes). This means +that regardless of the chosen transport, all incoming and outgoing packets are consistently sent through a single +Session. + ##### Broadcasting and neighborhood **What is a Neighborhood?** From b0782c02f146ae7cb31b043fa73a113ad0743269 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Wed, 30 Oct 2024 17:33:27 +0100 Subject: [PATCH 25/28] Introduction word to neighhood concept --- arch-snapshot/arch.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index 4b90d3d..532200c 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -931,7 +931,7 @@ Different transport types can be utilized to send messages, as explained in the channels chapter](#reliable-unreliable-and-transfers-channels). In the Hybrid Net, reliable and transfer transport types are distinguished by using separate TCP connections. This separation ensures that independent sender buffers are maintained, preventing messages in one channel from being blocked by messages in the other. Each transport type -has its own single TCP connection. This means that independent transfer still can interfere with each other. +has its own single TCP connection. This means that independent transfers still can interfere with each other. The sender can also opt to use the unreliable transport, where GSB messages are sent directly as `Forward` packets without message fragmentation. A key implication of this is that large GSB messages could exceed the Maximum @@ -943,13 +943,18 @@ Session. ##### Broadcasting and neighborhood +Naively broadcasting information, where each Node contacts every other Node, poses significant scalability limitations. +Thus, a more efficient method of information dissemination is necessary. Hybrid Net's implementation draws inspiration +from the [Kademlia algorithm](https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf), which utilizes distance metrics to minimize the number of control messages Nodes +need to exchange for discovery. While Kademlia serves a different purpose, Hybrid Net adapts its principles +by introducing the concept of a neighborhood. + **What is a Neighborhood?** -Before diving into broadcasting, it's essential to understand the concept of a neighborhood. In a network, a -neighborhood is a subset of Nodes that are considered closest to a given Node based on an abstract metric. Each Node -has its own neighborhood. This metric doesn't necessarily reflect the real-world proximity of Nodes. For instance, -two Nodes on opposite sides of the globe could be neighbors, while two Nodes within the same physical network may be -too distant in terms of this metric to be in the same neighborhood. +A neighborhood is a subset of Nodes that are considered closest to a given Node based on an abstract metric. +Each Node has its own neighborhood. This metric doesn't necessarily reflect the real-world proximity of Nodes. +For instance, two Nodes on opposite sides of the globe could be neighbors, while two Nodes within the same physical +network may be too distant in terms of this metric to be in the same neighborhood. In peer-to-peer networks, the concept of Node distance is often used for efficient Node discovery. However, since the current Golem network layer relies on the Relay server for Node discovery, neighborhoods don't serve that From 4d2a582e76d2a1406ec1899f569fe36ea6afbf61 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Wed, 30 Oct 2024 17:36:09 +0100 Subject: [PATCH 26/28] Apply correct broadcast naming semantic --- arch-snapshot/arch.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index 532200c..dde2512 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -970,9 +970,9 @@ blocks for these processes. **Broadcasting** -Hybrid Net implements local broadcast operation that sends message to the nearest neighborhood of the Node. To query -its neighbors, a Node can send a `Neighborhood` request to the Relay server. The Relay server then responds with a -list of Nodes that are closest to the querying Node, based on a predefined metric. +HybridNet implements the broadcast operation via sending broadcast messages to the nearest neighborhood of the Node. +To query its neighbors, a Node can send a `Neighborhood` request to the Relay server. The Relay server then responds +with a list of Nodes that are closest to the querying Node, based on a predefined metric. After receiving the list of neighbors, the Node attempts to establish Sessions with them, as described in the chapter on [communication](#establishing-sessions-between-nodes). The neighborhood algorithm does not From 25c1556e23eea92f37db316822c22501ed57c8c8 Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Thu, 31 Oct 2024 17:53:40 +0100 Subject: [PATCH 27/28] Remove duplicates after rebase --- arch-snapshot/arch.md | 40 ---------------------------------------- 1 file changed, 40 deletions(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index dde2512..557d351 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -589,32 +589,6 @@ block-beta Address2["/market/protocol/mk1/discovery/offers/Get"] ``` -The Net module follows specific GSB address naming conventions to enable cooperation with other modules. Addresses -prefixed with `/net/{NodeId}` are reserved for the Net module, where it listens for incoming messages and forwards -them to the Golem Network. Conversely, addresses starting with `/public/...` are available for yagna modules to expose -public methods that can be called from other Nodes. - -When the Net module receives a local incoming message, it extracts the NodeId from the address prefix and uses it to -forward the message into the Golem Network. On the receiving end, messages coming from the Network are processed, -and the address is checked to extract the NodeId. If the NodeId belongs to the recipient Node, the address is routed to -the appropriate GSB handler registered under the `/public/...` address. - -```mermaid -block-beta - columns 2 - Prefix{{"Prefix"}} - Address{{"Address"}} - - Prefix1["/net/0x467ab03ac10877d0ccff89fac547a4ce8aa0cc5e"] - Address1["/market/protocol/mk1/discovery/offers/Get"] - - arrow1<["Translate"]>(down) - space - - Prefix2["/public"] - Address2["/market/protocol/mk1/discovery/offers/Get"] -``` - ##### Broadcasting Message broadcasting in the Net module is organized around the concept of 'topics,' which can be thought of as @@ -650,20 +624,6 @@ The Net module must be able to handle messages sent to and from any of these ide identification, refer to the chapter about the [identity module](#identity). This section focuses solely on the Net module interface. -In addition to the GSB endpoints bound to the `/net/{NodeId}` prefix, as described in the [Address Translation -chapter](#address-translation), there is another prefix: `/from/{LocalId}/to/{RemoteId}`. This enables messages to -be sent from a specific identity on one Node to a specific identity on a remote Node. - -Another important aspect is that the Net module always checks if the target identity belongs to the local Node. If -it does, the message is routed back to the local GSB instead of being sent over the network. This mechanism allows -GSB calls to be handled uniformly by the calling code, regardless of whether the target is local or remote. - -Each Golem Node can have multiple identities, with one of them (the default identity) used to identify the Node -within the network. However, operations on a Golem Node can also be performed in the context of secondary identities. -The Net module must be able to handle messages sent to and from any of these identities. For more information on -identification, refer to the chapter about the [identity module](#identity). This section focuses solely on the Net -module interface. - In addition to the GSB endpoints bound to the `/net/{NodeId}` prefix, as described in the [GSB prefix mappings](#gsb-prefix-mappings), there is another prefix: `/from/{LocalId}/to/{RemoteId}`. This enables messages to be sent from a specific identity on one Node to a specific identity on a remote Node. From 6015150dde1ed59a3e83299711266d0336a44b8f Mon Sep 17 00:00:00 2001 From: "nieznany.sprawiciel" Date: Thu, 31 Oct 2024 17:57:59 +0100 Subject: [PATCH 28/28] Make transfer transport type description more precise --- arch-snapshot/arch.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch-snapshot/arch.md b/arch-snapshot/arch.md index 557d351..e4da0e5 100644 --- a/arch-snapshot/arch.md +++ b/arch-snapshot/arch.md @@ -891,7 +891,9 @@ Different transport types can be utilized to send messages, as explained in the channels chapter](#reliable-unreliable-and-transfers-channels). In the Hybrid Net, reliable and transfer transport types are distinguished by using separate TCP connections. This separation ensures that independent sender buffers are maintained, preventing messages in one channel from being blocked by messages in the other. Each transport type -has its own single TCP connection. This means that independent transfers still can interfere with each other. +has its own single TCP connection. If multiple components within the same process need to use the transfer channel, +they will share a single TCP connection, competing for bandwidth. As there is no fair scheduling mechanism, one +component can easily starve another by producing messages at a sufficiently high rate. The sender can also opt to use the unreliable transport, where GSB messages are sent directly as `Forward` packets without message fragmentation. A key implication of this is that large GSB messages could exceed the Maximum