Add scalable gossip library #1546
Conversation
aeyakovenko
added
work in progress
|
What I see here is 3 independent uses of crdts all thrown into one structure awkwardly named |
|
@garious its just not type agnostic. This actually implements a crdt with a merge strategy for all the different Value types. |
aeyakovenko
force-pushed
the
gossip_cleanup
branch
from
0f77d8a to
14574bb
Compare
|
@garious what do you think about this version? i got rid of the |
|
I don't like the approach of mixed the data type into the data structure and justifying it only by saying that it is cumbersome to do otherwise. To me that suggests you don't fully grok the data structure and that this implementation, like the one in cluster_info.rs, still shoots from the hip. I'd think that as soon as that clean separation existed, this would be implemented in two modules: one with a data structure and that references its relationship is a particular CRDT; one with the data that implements some Replica trait and describes how certain operations commute and so it can be used by a state-based CRDT. |
|
@garious can you show me an example of a clean heterogeneous type data structure in any language. once you figure that out, you can easily update this structure to that :) |
aeyakovenko
force-pushed
the
gossip_cleanup
branch
from
5fd37cd to
707dfe1
Compare
aeyakovenko
force-pushed
the
gossip_cleanup
branch
from
f0f5308 to
3841f2d
Compare
aeyakovenko
removed
noCI
aeyakovenko
force-pushed
the
gossip_cleanup
branch
from
3841f2d to
8a58322
Compare
Separate the data storage and merge strategy from the newtwork IO boundary. Implement an eager push overlay for transporting recent messages. Simulation shows fast convergance with 20k nodes.
aeyakovenko
requested review from
mvines and
pgarg66
and removed request for
pgarg66
aeyakovenko
force-pushed
the
gossip_cleanup
branch
from
e8707d9 to
9939f2f
Compare
|
@mvines this seems to work! i would like to merge before i get another big set of conflicts. |
|
@rob-solana / @carllin - please review for content too. You guys are more familiar with this part of the code than I |
src/crds_gossip_pull.rs
Outdated
| @@ -0,0 +1,378 @@ | |||
| //! Crds Gossip Pull overlay | |||
| //! This module implements the anti-entropy protocol for the newwork. | |||
src/crds_gossip_pull.rs
Outdated
| //! | ||
| //! The basic strategy is as follows: | ||
| //! 1. Construct a bloom filter of the local data set | ||
| //! 2. Randomly ask a node on the network for data that is is not contained in the bloom filter. |
src/crds_gossip_pull.rs
Outdated
| //! 1. Construct a bloom filter of the local data set | ||
| //! 2. Randomly ask a node on the network for data that is is not contained in the bloom filter. | ||
| //! | ||
| //! Bloom filters have a false positive rate. Each requests uses a differnet bloom filter |
There was a problem hiding this comment.
"differnet" -- heh, we gotta get you an editor with spelcheck!
src/crds_gossip_pull.rs
Outdated
| /// time when a request to `from` was initiated | ||
| /// This is used for weighted random selection durring `new_pull_request` | ||
| /// It's important to use the local nodes request creation time as the weight | ||
| /// instaad of the response received time otherwise failed nodes will increase their weight. |
src/crds_gossip_push.rs
Outdated
| //! | ||
| //! Main differences are: | ||
| //! 1. There is no `max hop`. Messages are signed with a local wallclock. If they are outside of | ||
| //! the local nodes wallclock window they are droped silently. |
…1506) (#1546) consensus: make shallow threshold checks log only (#1506) * consensus: make shallow threshold checks log only * pr feedback: comment, make check more readable (cherry picked from commit 6859d65) Co-authored-by: Ashwin Sekar <[email protected]>
…olana-labs#1506) (solana-labs#1546) consensus: make shallow threshold checks log only (solana-labs#1506) * consensus: make shallow threshold checks log only * pr feedback: comment, make check more readable (cherry picked from commit 6859d65) Co-authored-by: Ashwin Sekar <[email protected]>
Implements a scalable gossip protocol.