ICMP message-routing gossip #304

rphmeier · 2019-06-27T13:16:27Z

A description of the protocol can be found at https://research.web3.foundation/en/latest/polkadot/networking/4-icmp/ as well as in the module documentation of gossip.rs and message_routing.rs.

The goal is to create an everyone-sees-everything gossip protocol for routing inter-chain message queues. Each node will gossip only the queues it perceives as un-routed w.r.t. their best chain heads, and only send any given queue to peers who have some overlap in those chain heads - where the overlap's perspective of un-routed queues contains that queue.

This makes the protocol anti DoS-able and resilient against arbitrarily large Sybil attack, enabling us to open the gossip up to fishermen and collators instead of limiting it to validators.

network/src/gossip/attestation.rs

ltfschoen · 2019-07-31T02:02:31Z

network/src/gossip/attestation.rs

+		}
+	}
+
+	/// whether it's allowed to send a statement to a peer with given knowledge


Suggested change

/// whether it's allowed to send a statement to a peer with given knowledge

/// Whether it's allowed to send a statement to a peer with given knowledge

network/src/gossip/attestation.rs

network/src/gossip/message_routing.rs

+/// A view of which queue roots are current for a given set of leaves.
+pub struct View {
+	leaves: LeavesVec,
+	leaf_topics: HashMap<Hash, HashSet<Hash>>, // leaf_hash -> { topics }


ltfschoen · 2019-07-31T02:11:03Z

network/src/gossip/message_routing.rs

+pub struct View {
+	leaves: LeavesVec,
+	leaf_topics: HashMap<Hash, HashSet<Hash>>, // leaf_hash -> { topics }
+	expected_queues: HashMap<Hash, Hash>, // topic -> queue-root


Could we declare and then import as necessary some type aliases like the following to avoid having to add comments to describe the implementation:

pub type ExpectedQueueRootHash = T::Hash; pub type ExpectedQueueTopicHash = T::Hash;

Suggested change

expected_queues: HashMap<Hash, Hash>, // topic -> queue-root

expected_queues: HashMap<ExpectedQueueTopicHash, ExpectedQueueRootHash>,

network/src/gossip/message_routing.rs

ltfschoen

various nitpicks

Co-Authored-By: Luke Schoen <[email protected]>

…kadot into rh-icmp-gossip-primitives

gavofyork · 2019-08-15T13:10:00Z

availability-store/src/lib.rs

+				encoded.clone(),
+			);
+
+			tx.put_vec(


So this is storing the message data ready for some later recall under some circumstances? If those circumstances are not already documented, maybe put them here. If they are then a pointer to them somewhere near here would be most useful for anyone coming to this code.

gavofyork · 2019-08-15T13:12:15Z

collator/src/lib.rs

+								relay_parent,
+								targets,
+								collation,
+								extrinsic,


what is this change for?

gavofyork · 2019-08-15T15:03:00Z

network/src/gossip/attestation.rs

+		self.live_sessions.iter().take(MAX_CHAIN_HEADS).map(|(p, _)| p.clone())
+	}
+
+	/// Note a new session.


useless docs. function level docs should generally try to include:

typical context or situation in which the call is expected to be made

any expected preconditions to calling

what it is meant to do

how it is meant to interact with other functions

clear description of each argument

gavofyork · 2019-08-15T15:05:42Z

network/src/gossip/attestation.rs

+	}
+
+	/// Prune old sessions that fail the leaf predicate.
+	pub(super) fn prune_old_sessions<F: Fn(&Hash) -> bool>(&mut self, is_leaf: F) {


again, not super clear what this is meant to do. "sessions" clashes with runtime terminology and is_leaf's meaning is unclear (should it be renamed to keep_leaf?).

gavofyork · 2019-08-15T15:06:20Z

network/src/gossip/attestation.rs

+		self.topics.retain(|_, v| live_sessions.iter().find(|(p, _)| p == v).is_some());
+	}
+
+	/// Whether a topic is live.


what does "live" mean in this context? is it defined in this file? (i don't see a definition anywhere.)

gavofyork · 2019-08-15T15:07:10Z

network/src/gossip/attestation.rs

+	}
+
+
+	/// Validate an attestation statement of some kind.


fuller docs needed (as above). what does it mean to "validate" in this context?

network/src/gossip/message_routing.rs

* core logic for ICMP gossip * refactor gossip to make more extension friendly * move files aroun * extract attestation-gossip logic to its own module * message validation and broadcast logic * fix upstream crates' compilation * add a test * another test for overlapping * Some grammar and phrasing tweaks Co-Authored-By: Luke Schoen <[email protected]> * add since parameter to ingress runtime API * broadcast out known unrouted message queues * fix compilation of service and collator * remove useless index_mapping * some tests for icmp propagation * fix decoding bug and test icmp queue validation * simplify engine-id definition Co-Authored-By: Sergei Pepyakin <[email protected]> * address some grumbles * some cleanup of old circulation code * give network a handle to extrinsic store on startup * an honest collator ensures data available as well * address some grumbles * add docs; rename the attestation session to "leaf work" * module docs * move gossip back to gossip.rs * clean up and document attestation-gossip a bit * some more docs on the availability store * store all outgoing message queues in the availability store * filter `Extrinsic` out of validation crate * expunge Extrinsic from network * expunge Extrinsic from erasure-coding * expunge Extrinsic from collator * expunge from adder-collator * rename ExtrinsicStore to AvailabilityStore everywhere * annotate and clean up message-routing tests

* Update to latest Substrate (#407) * Update to latest Substrate * Fix main.rs * Update substrate master (#411) * in progress impl * im_online authorityid * fix * fix * use polkadot-master * trigger CI * trigger CI * fix removal * storage reorganize included * lock version * remove unused * increment spec version * ICMP message-routing gossip (#304) * core logic for ICMP gossip * refactor gossip to make more extension friendly * move files aroun * extract attestation-gossip logic to its own module * message validation and broadcast logic * fix upstream crates' compilation * add a test * another test for overlapping * Some grammar and phrasing tweaks Co-Authored-By: Luke Schoen <[email protected]> * add since parameter to ingress runtime API * broadcast out known unrouted message queues * fix compilation of service and collator * remove useless index_mapping * some tests for icmp propagation * fix decoding bug and test icmp queue validation * simplify engine-id definition Co-Authored-By: Sergei Pepyakin <[email protected]> * address some grumbles * some cleanup of old circulation code * give network a handle to extrinsic store on startup * an honest collator ensures data available as well * address some grumbles * add docs; rename the attestation session to "leaf work" * module docs * move gossip back to gossip.rs * clean up and document attestation-gossip a bit * some more docs on the availability store * store all outgoing message queues in the availability store * filter `Extrinsic` out of validation crate * expunge Extrinsic from network * expunge Extrinsic from erasure-coding * expunge Extrinsic from collator * expunge from adder-collator * rename ExtrinsicStore to AvailabilityStore everywhere * annotate and clean up message-routing tests

rphmeier added 4 commits June 27, 2019 11:49

core logic for ICMP gossip

207e5b6

refactor gossip to make more extension friendly

360c729

move files aroun

9922f02

extract attestation-gossip logic to its own module

0125f18

rphmeier added the A3-in_progress Pull request is in progress. No review needed at this stage. label Jun 27, 2019

rphmeier added 5 commits July 24, 2019 11:45

Merge branch 'master' into rh-icmp-gossip-primitives

5e25763

message validation and broadcast logic

3a4bd99

fix upstream crates' compilation

9e76e36

add a test

65bbe33

another test for overlapping

e9e6032

ltfschoen reviewed Jul 31, 2019

View reviewed changes

network/src/gossip/attestation.rs Show resolved Hide resolved

ltfschoen reviewed Jul 31, 2019

View reviewed changes

network/src/gossip/attestation.rs Outdated Show resolved Hide resolved

ltfschoen reviewed Jul 31, 2019

View reviewed changes

network/src/gossip/attestation.rs Show resolved Hide resolved

ltfschoen reviewed Jul 31, 2019

View reviewed changes

network/src/gossip/message_routing.rs Outdated Show resolved Hide resolved

ltfschoen reviewed Jul 31, 2019

View reviewed changes

network/src/gossip/message_routing.rs Outdated Show resolved Hide resolved

ltfschoen reviewed Jul 31, 2019

View reviewed changes

rphmeier and others added 10 commits August 2, 2019 13:17

Some grammar and phrasing tweaks

a192da1

Co-Authored-By: Luke Schoen <[email protected]>

Merge branch 'master' into rh-icmp-gossip-primitives

8d4e67b

add since parameter to ingress runtime API

63f26b0

broadcast out known unrouted message queues

ec7d9de

fix compilation of service and collator

bdc213e

Merge branch 'rh-icmp-gossip-primitives' of github.com:paritytech/pol…

cfa47b8

…kadot into rh-icmp-gossip-primitives

remove useless index_mapping

b38b712

Merge branch 'master' into rh-icmp-gossip-primitives

727b92c

some tests for icmp propagation

6a9269d

fix decoding bug and test icmp queue validation

ced6186

rphmeier removed the A3-in_progress Pull request is in progress. No review needed at this stage. label Aug 12, 2019

gavofyork reviewed Aug 15, 2019

View reviewed changes

collator/src/lib.rs Outdated

relay_parent,

targets,

collation,

extrinsic,

Copy link

Member

gavofyork Aug 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this change for?

gavofyork reviewed Aug 15, 2019

View reviewed changes

montekki reviewed Aug 15, 2019

View reviewed changes

network/src/gossip/message_routing.rs Outdated Show resolved Hide resolved

network/src/gossip/message_routing.rs Outdated Show resolved Hide resolved

network/src/gossip/message_routing.rs Show resolved Hide resolved

clean up and document attestation-gossip a bit

02ecf76

rphmeier added A3-in_progress Pull request is in progress. No review needed at this stage. and removed A0-please_review Pull request needs code review. labels Aug 15, 2019

some more docs on the availability store

5ccf536

montekki reviewed Aug 15, 2019

View reviewed changes

network/src/gossip/message_routing.rs Show resolved Hide resolved

rphmeier added 8 commits August 15, 2019 18:35

store all outgoing message queues in the availability store

acc36af

filter Extrinsic out of validation crate

d736ea3

expunge Extrinsic from network

bded179

expunge Extrinsic from erasure-coding

ad31039

expunge Extrinsic from collator

550c4fd

expunge from adder-collator

7a82f33

rename ExtrinsicStore to AvailabilityStore everywhere

e900307

annotate and clean up message-routing tests

093a912

rphmeier added A0-please_review Pull request needs code review. and removed A3-in_progress Pull request is in progress. No review needed at this stage. labels Aug 15, 2019

Merge branch 'master' into rh-icmp-gossip-primitives

bba4c5a

rphmeier merged commit 03cfa5e into master Aug 29, 2019

rphmeier deleted the rh-icmp-gossip-primitives branch August 29, 2019 09:50

gui1117 mentioned this pull request Sep 3, 2019

Backport updates of substrate #415

Merged

syuukawa mentioned this pull request Sep 13, 2019

001-平行链中继链和Birdge SubstrateOrg/Team1#11

Open

andresilva mentioned this pull request Oct 21, 2019

Only register one gossip validator for full nodes #487

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ICMP message-routing gossip #304

ICMP message-routing gossip #304

rphmeier commented Jun 27, 2019 •

edited

Loading

ltfschoen Jul 31, 2019

This comment was marked as resolved.

This comment was marked as resolved.

ltfschoen Jul 31, 2019

ltfschoen left a comment

gavofyork Aug 15, 2019

gavofyork Aug 15, 2019

gavofyork Aug 15, 2019 •

edited

Loading

gavofyork Aug 15, 2019

gavofyork Aug 15, 2019

gavofyork Aug 15, 2019

	/// whether it's allowed to send a statement to a peer with given knowledge
	/// Whether it's allowed to send a statement to a peer with given knowledge

	expected_queues: HashMap<Hash, Hash>, // topic -> queue-root
	expected_queues: HashMap<ExpectedQueueTopicHash, ExpectedQueueRootHash>,

ICMP message-routing gossip #304

ICMP message-routing gossip #304

Conversation

rphmeier commented Jun 27, 2019 • edited Loading

ltfschoen Jul 31, 2019

Choose a reason for hiding this comment

This comment was marked as resolved.

This comment was marked as resolved.

ltfschoen Jul 31, 2019

Choose a reason for hiding this comment

ltfschoen left a comment

Choose a reason for hiding this comment

gavofyork Aug 15, 2019

Choose a reason for hiding this comment

gavofyork Aug 15, 2019

Choose a reason for hiding this comment

gavofyork Aug 15, 2019 • edited Loading

Choose a reason for hiding this comment

gavofyork Aug 15, 2019

Choose a reason for hiding this comment

gavofyork Aug 15, 2019

Choose a reason for hiding this comment

gavofyork Aug 15, 2019

Choose a reason for hiding this comment

rphmeier commented Jun 27, 2019 •

edited

Loading

gavofyork Aug 15, 2019 •

edited

Loading