-
Notifications
You must be signed in to change notification settings - Fork 60
fix: re-enable ensuring queries run along disjoint paths #371
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@achingbrain should also have a look
src/peer-routing/index.ts
Outdated
@@ -235,8 +235,6 @@ export class PeerRouting implements Initializable { | |||
} | |||
|
|||
for await (const event of this.queryManager.run(key, tablePeers, getCloserPeersQuery, options)) { | |||
yield event |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why has this been removed? Yielding intermediate events is useful for query diagnostics and tracing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From my understanding, getClosestPeers should return only kBucketSize closest peers. This implementation currently also returns multiple "in between" peers instead coming from events other than "FINAL_PEER".
I removed this so that only final peers are returned. It could be there is a misunderstanding on my part. If that's the case, I can add it back.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would expect this function to yield the kBucketSize
sorted peers, but it's yielding a lot more than expected (not all are FINAL_PEER
), and not necessarily sorted (because it merges events yielded by both lan
and wan
dhts).
This feature is called disjoint paths - it's in the S/Kademlia paper and this impl used to support it. Notably the go KAD-DHT implementation does not have this feature which is why it was removed from this repo, though I notice the rust implementation does support it so there's no harm in re-adding it here, and is certainly worth doing if it makes the queries run faster. Thanks for opening this! |
The final thing to do to get this in is to write some tests. For an example you can see how we create predictable network topologies in test/query.spec.js and run a query which records the peers it passed through to get to the final value. You should be able to validate in a test that queries stop when passing through the same node twice. |
Thanks! |
🎉 This PR is included in version 3.0.6 🎉 The release is available on: Your semantic-release bot 📦🚀 |
I noticed that during the getClosestPeers operation, the query paths might overlap and the same nodes will be queried multiple times.
There is no need to query a node if it has been queried during the execution of another query path, so the
peersSeen
set should be shared among different query paths belonging to the same query.