go/worker/common: Refresh node descriptors mid-epoch #2584
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #2584 +/- ##
=========================================
- Coverage 63.47% 63.38% -0.1%
=========================================
Files 357 359 +2
Lines 33682 33841 +159
=========================================
+ Hits 21381 21449 +68
- Misses 9693 9764 +71
- Partials 2608 2628 +20
Continue to review full report at Codecov.
|
kostko
force-pushed
the
kostko/feature/group-node-refresh
branch
7 times, most recently
from
7f3f147 to
c607f08
Compare
Yawning
approved these changes
Jan 24, 2020
kostko
force-pushed
the
kostko/feature/group-node-refresh
branch
8 times, most recently
from
f11ae03 to
08bfcc4
Compare
Yawning
reviewed
Jan 28, 2020
kostko
force-pushed
the
kostko/feature/group-node-refresh
branch
from
08bfcc4 to
d52b3da
Compare
Previously node descriptors were only refreshed on an epoch transition which meant that any later updates were ignored. This caused stale RAKs to stay in effect when runtime restarts happened. Enabling mid-epoch refresh also makes having more ephemeral keys easier.
Commit 2fdfd28 introduced a possibility of the storage round syncing process to deadlock when an unfortunate series of events occur in specific order. Assume the following happens, in order: 1. Round X has just completed syncing and has been applied, so lastFullyAppliedRound has been updated to X and all metadata about round X has been removed from syncingRounds and hashCache. Since the round has not yet been finalized, cachedLastRound still points to X-1. 2. A new incoming block for round X+1 appears. 3. The code checks what needs to be synced and starts with cachedLastRound which still points to round X-1 (because round X has not yet been applied). Because sync metadata of round X have been removed, it assumes that round X is not yet synced so it starts syncing it by queuing sync requests and adding new metadata to the top of the syncingRounds heap. 4. Round X is finalized, so cachedLastRound is updated to X. Since the top of the sync management loop checks if lastFullyAppliedRound + 1 is at the top of the heap this leads to a deadlock. The top item contains metadata for round X while the management loop is waiting for X+1. This commit fixes the issue by using lastFullyAppliedRound in the incoming block handler instead of cachedLastRound.
Also use new generalized interfaces for managing gRPC connections to committee members (with support for mid-epoch refresh).
kostko
force-pushed
the
kostko/feature/group-node-refresh
branch
from
d52b3da to
cc98774
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1794.
Fixes #2193
Previously node descriptors were only refreshed on an epoch transition which
meant that any later updates were ignored. This caused stale RAKs to stay in
effect when runtime restarts happened.
Enabling mid-epoch refresh also makes having more ephemeral keys easier.
TODO
NodeInfofrom commitment pool.