Beefy on-demand justifications as a custom RequestResponse protocol #12124

acatangiu · 2022-08-26T16:57:47Z

Description

Run a dedicated BEEFY justifications sync protocol as a libp2p RequestResponse protocol - where node can request BEEFY justifications for certain blocks from connected peers, and also answer any requests it gets.

PR defines BEEFY own RequestResponse sub-protocol and adds support for:

handling incoming requests: BeefyJustifsRequestHandler in incoming_requests_handler.rs
requesting justifications from peers: OnDemandJustificationsEngine in outgoing_requests_engine.rs

BeefyJustifsRequestHandler

Run as independent async task, accepts incoming requests, decodes and validates request for BEEFY justification for block b.
It then uses client backend to retrieve justification if available, encode and provide it as custom protocol response.

OnDemandJustificationsEngine

Simple async state machine that be either:

"idle" - no on-demand justifications required
"requesting justification for block b" - self driving future that sequentially sends requests to all relevant peers until one provides valid justification or no more peers available. relevant peers are those seen (through gossip) voting for block numbers >= b, meaning they should have justification for b.

main worker/voter

Requests justifications from OnDemandJustificationsEngine for every mandatory block it's currently working/voting/blocked on.
Drives the OnDemandJustificationsEngine inner future in its main loop.

Tests

Added integration test that verifies all components and full workflow:

1 out of 4 voters is held behind so that it misses voting for a number of mandatory blocks,
when started, late voter must get justifications for past mandatory blocks and can only do so through on-demand requests,
other 3 voters need to provide on-demand justifications responses for late voter to catch up,
by end of test, late voter is up-to-date and actively participating in voting.

Fixes

Fixes #12093 (more details there).

polkadot companion: paritytech/polkadot#6035

…ustom-sync-protocol

Fix corner case where voter gets a single burst of finality notifications just when it starts. The notification stream was consumed by "wait_for_pallet" logic, then main loop would subscribe to finality notifications, but by that time some notifications might've been lost. Fix this by subscribing the main loop to notifications before waiting for pallet to become available. Share the same stream with the main loop so that notifications for blocks before pallet available are ignored, while _all_ notifications after pallet available are processed. Add regression test for this. Signed-off-by: acatangiu <[email protected]>

…blocks

…ustom-sync-protocol

…oter

…ustom-sync-protocol

altonen

Left some minor comments but looks good!

client/beefy/src/communication/peers.rs

client/beefy/src/communication/request_response/incoming_requests_handler.rs

client/beefy/src/communication/request_response/outgoing_requests_engine.rs

client/beefy/src/communication/request_response/incoming_requests_handler.rs

client/beefy/src/communication/request_response/outgoing_requests_engine.rs

acatangiu

Thanks for the review! I've addressed all comments 👍

client/beefy/src/communication/request_response/incoming_requests_handler.rs

client/beefy/src/communication/request_response/outgoing_requests_engine.rs

…ustom-sync-protocol

client/beefy/src/communication/mod.rs

dmitry-markin · 2022-10-03T11:25:12Z

client/beefy/src/communication/request_response/outgoing_requests_engine.rs

+	pub async fn next(&mut self) -> Option<BeefyVersionedFinalityProof<B>> {
+		let (peer, block, resp) = match &mut self.state {
+			State::Idle(pending) => {
+				let _ = pending.next().await;


A noobs async question: does this rely on next() being polled in a select! loop with other futures, otherwise we'll get stuck here forever?

This whole function is async (as seen in pub async fn signature). This means it cannot be (directly) called from a normal sync context - the compiler will complain (this is because async functions actually translates to sync functions returning impl Future - a pretty good article on this here).

.await does indeed mean "wait here until this is ready", but it can only be called inside an async function, meaning "wait here" is not "block here", but rather "do something else until this is ready, then come back here".
The way this happens under the hood is by turning this whole function into a state machine where each await is a transition from one state to another - a pretty good explanation here.

So to answer your question, no, this function does not rely on it being called in a select! loop, the only requirements on it are the ones enforced by the compiler: the function needs to be called from either another async context/fn or must be directly called by an "asynchronous runtime".

@acatangiu thanks a lot for the detailed explanation!

acatangiu · 2022-10-03T12:02:13Z

bot merge

paritytech-processbot · 2022-10-03T12:02:21Z

Error: Statuses failed for b904741

…ustom-sync-protocol

acatangiu · 2022-10-03T12:54:06Z

bot merge

paritytech-processbot · 2022-10-03T12:54:14Z

Waiting for commit status.

bkchr

I know that I'm very late to the party. I still left some comments on stuff that could be improved. Sorry!

client/beefy/src/lib.rs

client/beefy/src/communication/request_response/outgoing_requests_engine.rs

client/beefy/src/communication/mod.rs

acatangiu · 2022-10-04T10:44:34Z

I know that I'm very late to the party. I still left some comments on stuff that could be improved. Sorry!

Better late than never 😛 thanks for the improvement ideas! Opened #12414 for them.

@koute

* master: (42 commits) Adapt `pallet-contracts` to WeightV2 (#12421) Improved election pallet testing (#12327) Bump prost to 0.11+ (#12419) Use saturating add for alliance::disband witness data (#12418) [Fix] Rename VoterBagsList -> VoterList to match pdot (#12416) client/beefy: small code improvements (#12414) BEEFY: Simplify hashing for pallet-beefy-mmr (#12393) Add @koute to `docs/CODEOWNERS` and update stale paths (#12408) docs/CODEOWNERS: add @acatangiu as MMR owner (#12406) Remove unnecessary Clone trait bounds on CountedStorageMap (#12402) Fix `Weight::is_zero` (#12396) Beefy on-demand justifications as a custom RequestResponse protocol (#12124) Remove contracts RPCs (#12358) pallet-mmr: generate historical proofs (#12324) unsafe_pruning flag removed (#12385) Carry over where clauses defined in Config to Call and Hook (#12388) Properly set the max proof size weight on defaults and tests (#12383) BEEFY: impl TypeInfo for SignedCommitment (#12382) bounding staking: `BoundedElectionProvider` trait (#12362) New Pallet: Root offences (#11943) ...

…aritytech#12124) * client/beefy: create communication module and move gossip there * client/beefy: move beefy_protocol_name module to communication * client/beefy: move notification module under communication * client/beefy: add incoming request_response protocol handler * client/beefy: keep track of connected peers and their progress * client/beefy: add logic for generating Justif requests * client/beefy: cancel outdated on-demand justification requests * try Andre's suggestion for JustificationEngine * justif engine add justifs validation * client/beefy: impl OnDemandJustificationsEngine async next() * move beefy proto name test * client/beefy: initialize OnDemandJustificationsEngine * client/tests: allow for custom req-resp protocols * client/beefy: on-demand-justif: implement simple peer selection strategy * client/beefy: fix voter initialization Fix corner case where voter gets a single burst of finality notifications just when it starts. The notification stream was consumed by "wait_for_pallet" logic, then main loop would subscribe to finality notifications, but by that time some notifications might've been lost. Fix this by subscribing the main loop to notifications before waiting for pallet to become available. Share the same stream with the main loop so that notifications for blocks before pallet available are ignored, while _all_ notifications after pallet available are processed. Add regression test for this. Signed-off-by: acatangiu <[email protected]> * client/beefy: make sure justif requests are always out for mandatory blocks * client/beefy: add test for on-demand justifications sync * client/beefy: tweak main loop event processing order * client/beefy: run on-demand-justif-handler under same async task as voter * client/beefy: add test for known-peers * client/beefy: reorg request-response module * client/beefy: add issue references for future work todos * client/beefy: consolidate on-demand-justifications engine state machine Signed-off-by: acatangiu <[email protected]> * client/beefy: fix for polkadot companion * client/beefy: implement review suggestions * cargo fmt and clippy * fix merge damage * fix rust-doc * fix merge damage * fix merge damage * client/beefy: add test for justif proto name Signed-off-by: acatangiu <[email protected]>

acatangiu added 7 commits August 26, 2022 19:32

client/beefy: create communication module and move gossip there

46a4231

client/beefy: move beefy_protocol_name module to communication

cda48fc

client/beefy: move notification module under communication

e8157d7

client/beefy: add incoming request_response protocol handler

f9bf81b

client/beefy: keep track of connected peers and their progress

f68920e

client/beefy: add logic for generating Justif requests

5fc1bb7

client/beefy: cancel outdated on-demand justification requests

9f05d3a

acatangiu added A0-please_review Pull request needs code review. B0-silent Changes should not be mentioned in any release notes C1-low PR touches the given topic and has a low impact on builders. labels Aug 26, 2022

acatangiu requested review from andresilva and bkchr August 26, 2022 16:57

acatangiu self-assigned this Aug 26, 2022

acatangiu added 13 commits September 9, 2022 22:00

try Andre's suggestion for JustificationEngine

ac5c3a5

justif engine add justifs validation

e9ebea7

Merge branch 'master' of github.com:paritytech/substrate into beefy-c…

fbd4f57

…ustom-sync-protocol

client/beefy: impl OnDemandJustificationsEngine async next()

4ab5ab0

move beefy proto name test

88f8904

client/beefy: initialize OnDemandJustificationsEngine

b5bf047

client/tests: allow for custom req-resp protocols

2e47d2c

client/beefy: on-demand-justif: implement simple peer selection strategy

fdebcaa

Merge branch 'master' of github.com:paritytech/substrate into beefy-c…

9aedbfa

…ustom-sync-protocol

client/beefy: make sure justif requests are always out for mandatory …

a9c60df

…blocks

client/beefy: add test for on-demand justifications sync

d48e199

client/beefy: tweak main loop event processing order

3cb39f7

acatangiu force-pushed the beefy-custom-sync-protocol branch from 64be1f2 to 3cb39f7 Compare September 16, 2022 13:35

acatangiu added 3 commits September 16, 2022 16:40

Merge branch 'master' of github.com:paritytech/substrate into beefy-c…

1908941

…ustom-sync-protocol

Merge branch 'master' of github.com:paritytech/substrate into beefy-c…

d265845

…ustom-sync-protocol

client/beefy: run on-demand-justif-handler under same async task as v…

f76dd02

…oter

Merge branch 'master' of github.com:paritytech/substrate into beefy-c…

b226996

…ustom-sync-protocol

acatangiu mentioned this pull request Sep 21, 2022

Companion for substrate#12124: add BEEFY request response protocol paritytech/polkadot#6035

Merged

acatangiu requested a review from serban300 September 21, 2022 16:34

client/beefy: fix for polkadot companion

68854a8

altonen approved these changes Sep 22, 2022

View reviewed changes

acatangiu commented Sep 22, 2022

View reviewed changes

acatangiu added 2 commits September 22, 2022 13:56

client/beefy: implement review suggestions

683ddc5

cargo fmt and clippy

200cdf0

acatangiu force-pushed the beefy-custom-sync-protocol branch from fabd146 to 200cdf0 Compare September 23, 2022 13:21

acatangiu added 6 commits September 26, 2022 13:01

Merge branch 'master' of github.com:paritytech/substrate into beefy-c…

7a8950d

…ustom-sync-protocol

fix merge damage

447dc2d

fix rust-doc

65cc99e

Merge branch 'master' of github.com:paritytech/substrate into beefy-c…

9d6cd2f

…ustom-sync-protocol

fix merge damage

c68cbef

fix merge damage

9a6bccd

dmitry-markin approved these changes Oct 3, 2022

View reviewed changes

client/beefy: add test for justif proto name

b904741

Merge branch 'master' of github.com:paritytech/substrate into beefy-c…

fd50ee7

…ustom-sync-protocol

paritytech-processbot bot merged commit 9472af8 into paritytech:master Oct 3, 2022

bkchr reviewed Oct 4, 2022

View reviewed changes

acatangiu mentioned this pull request Oct 4, 2022

client/beefy: small code improvements #12414

Merged

acatangiu deleted the beefy-custom-sync-protocol branch December 13, 2022 08:39

AurevoirXavier mentioned this pull request Jan 10, 2023

To polkadot-v0.9.33 darwinia-network/darwinia-2.0#194

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Beefy on-demand justifications as a custom RequestResponse protocol #12124

Beefy on-demand justifications as a custom RequestResponse protocol #12124

acatangiu commented Aug 26, 2022 •

edited

Loading

altonen left a comment

acatangiu left a comment

dmitry-markin Oct 3, 2022

acatangiu Oct 3, 2022

dmitry-markin Oct 3, 2022 •

edited

Loading

acatangiu commented Oct 3, 2022

paritytech-processbot bot commented Oct 3, 2022

acatangiu commented Oct 3, 2022

paritytech-processbot bot commented Oct 3, 2022

bkchr left a comment

acatangiu commented Oct 4, 2022

Beefy on-demand justifications as a custom RequestResponse protocol #12124

Beefy on-demand justifications as a custom RequestResponse protocol #12124

Conversation

acatangiu commented Aug 26, 2022 • edited Loading

Description

BeefyJustifsRequestHandler

OnDemandJustificationsEngine

main worker/voter

Tests

Fixes

altonen left a comment

Choose a reason for hiding this comment

acatangiu left a comment

Choose a reason for hiding this comment

dmitry-markin Oct 3, 2022

Choose a reason for hiding this comment

acatangiu Oct 3, 2022

Choose a reason for hiding this comment

dmitry-markin Oct 3, 2022 • edited Loading

Choose a reason for hiding this comment

acatangiu commented Oct 3, 2022

paritytech-processbot bot commented Oct 3, 2022

acatangiu commented Oct 3, 2022

paritytech-processbot bot commented Oct 3, 2022

bkchr left a comment

Choose a reason for hiding this comment

acatangiu commented Oct 4, 2022

acatangiu commented Aug 26, 2022 •

edited

Loading

dmitry-markin Oct 3, 2022 •

edited

Loading