Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kad: Refactor FindNode query, keep K best results and add tests #114

Merged
merged 14 commits into from
May 24, 2024

Conversation

lexnv
Copy link
Collaborator

@lexnv lexnv commented May 21, 2024

This PR refactors the FindNode query to make it more robust.

  • Avoid panics on unwraps and unimplemented logic
  • Separate config immutable variables from main query logic
  • Simplifies logic of next_action method
  • Private methods for internal logic

It also fixes a bug where only the last peer ID of our results was updated.

  • we were registering the first 20 (repl factor) responses
  • whenever we got a new response (better) we tried to update just the peerID_ of the last entry

This had a few implications:

  • the first 19 entries are never updated with a better (closer) peer
  • the distance of the last entry was never updated. And because of this, we might end up in a situation where the record is updated with a furthest distance
// Presume: 
last_entry: distance = 10, peer = A

// Registering a new response
new_response: distance = 2, peer = B
-> last entry: distance = 10, peer = B

// Registering a new response
new_response: distance = 7, peer = C
-> last entry: distance = 10, peer = C

// we should've kept B as best

Testing Done

  • Completes if there's no further candidate to query
  • Fulfill number of parallel queries
  • Completes on the number of responses
  • Provides the closest response out of the batch
  • Provides the closest response when the closest peer is indirectly found and K factor is already fulfilled

Closes #100

cc @paritytech/networking

@lexnv lexnv self-assigned this May 21, 2024
Copy link
Collaborator

@dmitry-markin dmitry-markin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice PR! And good to see we have FIND_NODE tested now.


// always mark the peer as queried to prevent it getting queried again
self.queried.insert(peer.peer);

// TODO: could this be written in another way?
// TODO: only insert nodes from whom a response was received
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this TODO still relevant? It looks like nothing has changed in its respect.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont think this is addressed indeed, will briefly look over libp2p or create an issue if this may be too involved / not covered by our testing atm, thanks!

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While looking at this I realized a few things:

  • we were registering the first 20 (repl factor) responses
  • whenever we got a new response (better) we tried to update just the peerID_ of the last entry

This has a few implications:

  • the first 19 entries are never updated with a better (closer) peer
  • the distance of the last entry was never updated. And because of this, we might end up in a situation where the record is updated with a furthest distance
// Presume: 
last_entry: distance = 10, peer = A

// Registering a new response
new_response: distance = 2, peer = B
-> last entry: distance = 10, peer = B

// Registering a new response
new_response: distance = 7, peer = C
-> last entry: distance = 10, peer = C

// we should've kept B as best

In this case, we should have updated other records and only keep the window of the best 20 peers.

To mitigate this, if we receive a response better than the last entry:

  • insert the response with distance and peer to our btreemap, if and only if we didn't track that distance before
  • pop the last entry to keep only K entries

Have also added a test for this keep_k_best_results

@lexnv lexnv changed the title kad: Refactor FindNode query and add tests kad: Refactor FindNode query, keep K best results and add tests May 23, 2024

// The response received from the peer is closer than the furthest response.
if distance < furthest_distance {
// Update the entries only if the distance is not already present.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the drawback of overwriting the entry with this distance? I.e., there should be exactly one peer with such distance as it uniquely identifies the hash, so it should be safe to overwrite without checking.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I followed the libp2p here. However, I believe we can overwrite the entries without checking here, and we'll have a "fresher" report in the end. Will do that, thanks!

@lexnv lexnv merged commit e022410 into master May 24, 2024
8 checks passed
@lexnv lexnv deleted the lexnv/refactor-find-node branch May 24, 2024 10:56
@lexnv lexnv mentioned this pull request May 24, 2024
lexnv added a commit that referenced this pull request May 24, 2024
## [0.5.0] - 2023-05-24

This is a small patch release that makes the `FindNode` command a bit
more robst:

- The `FindNode` command now retains the K (replication factor) best
results.
- The `FindNode` command has been updated to handle errors and
unexpected states without panicking.

### Changed

- kad: Refactor FindNode query, keep K best results and add tests
([#114](#114))

## [0.4.0] - 2023-05-23

This release introduces breaking changes to the litep2p crate, primarily
affecting the `kad` module. Key updates include:

- The `GetRecord` command now exposes all peer records, not just the
latest one.
- A new `RecordType` has been introduced to clearly distinguish between
locally stored records and those discovered from the network.

Significant refactoring has been done to enhance the efficiency and
accuracy of the `kad` module. The updates are as follows:

- The `GetRecord` command now exposes all peer records.
- The `GetRecord` command has been updated to handle errors and
unexpected states without panicking.

Additionally, we've improved code coverage in the `kad` module by adding
more tests.

### Added

- Add release checklist
([#115](#115))
- Re-export `multihash` & `multiaddr` types
([#79](#79))
- kad: Expose all peer records of `GET_VALUE` query
([#96](#96))

### Changed

- multistream_select: Remove unneeded changelog.md
([#116](#116))
- kad: Refactor `GetRecord` query and add tests
([#97](#97))
- kad/store: Set memory-store on an incoming record for PutRecordTo
([#88](#88))
- multistream: Dialer deny multiple /multistream/1.0.0 headers
([#61](#61))
- kad: Limit MemoryStore entries
([#78](#78))
- Refactor WebRTC code
([#51](#51))
- Revert "Bring `rustfmt.toml` in sync with polkadot-sdk (#71)"
([#74](#74))
- cargo: Update str0m from 0.4.1 to 0.5.1
([#95](#95))

### Fixed

- Fix clippy  ([#83](#83))
- crypto: Don't panic on unsupported key types
([#84](#84))

---------

Signed-off-by: Alexandru Vasile <[email protected]>
github-merge-queue bot pushed a commit to paritytech/polkadot-sdk that referenced this pull request May 27, 2024
## [0.5.0] - 2023-05-24

This is a small patch release that makes the `FindNode` command a bit
more robst:

- The `FindNode` command now retains the K (replication factor) best
results.
- The `FindNode` command has been updated to handle errors and
unexpected states without panicking.

### Changed

- kad: Refactor FindNode query, keep K best results and add tests
([#114](paritytech/litep2p#114))

---------

Signed-off-by: Alexandru Vasile <[email protected]>
github-merge-queue bot pushed a commit to paritytech/polkadot-sdk that referenced this pull request May 27, 2024
## [0.5.0] - 2023-05-24

This is a small patch release that makes the `FindNode` command a bit
more robst:

- The `FindNode` command now retains the K (replication factor) best
results.
- The `FindNode` command has been updated to handle errors and
unexpected states without panicking.

### Changed

- kad: Refactor FindNode query, keep K best results and add tests
([#114](paritytech/litep2p#114))

---------

Signed-off-by: Alexandru Vasile <[email protected]>
lexnv added a commit to paritytech/polkadot-sdk that referenced this pull request May 27, 2024
This is a small patch release that makes the `FindNode` command a bit
more robst:

- The `FindNode` command now retains the K (replication factor) best
results.
- The `FindNode` command has been updated to handle errors and
unexpected states without panicking.

- kad: Refactor FindNode query, keep K best results and add tests
([#114](paritytech/litep2p#114))

---------

Signed-off-by: Alexandru Vasile <[email protected]>
lexnv added a commit to paritytech/polkadot-sdk that referenced this pull request May 27, 2024
This is a small patch release that makes the `FindNode` command a bit
more robst:

- The `FindNode` command now retains the K (replication factor) best
results.
- The `FindNode` command has been updated to handle errors and
unexpected states without panicking.

- kad: Refactor FindNode query, keep K best results and add tests
([#114](paritytech/litep2p#114))

---------

Signed-off-by: Alexandru Vasile <[email protected]>
hitchhooker pushed a commit to ibp-network/polkadot-sdk that referenced this pull request Jun 5, 2024
## [0.5.0] - 2023-05-24

This is a small patch release that makes the `FindNode` command a bit
more robst:

- The `FindNode` command now retains the K (replication factor) best
results.
- The `FindNode` command has been updated to handle errors and
unexpected states without panicking.

### Changed

- kad: Refactor FindNode query, keep K best results and add tests
([paritytech#114](paritytech/litep2p#114))

---------

Signed-off-by: Alexandru Vasile <[email protected]>
TarekkMA pushed a commit to moonbeam-foundation/polkadot-sdk that referenced this pull request Aug 2, 2024
## [0.5.0] - 2023-05-24

This is a small patch release that makes the `FindNode` command a bit
more robst:

- The `FindNode` command now retains the K (replication factor) best
results.
- The `FindNode` command has been updated to handle errors and
unexpected states without panicking.

### Changed

- kad: Refactor FindNode query, keep K best results and add tests
([paritytech#114](paritytech/litep2p#114))

---------

Signed-off-by: Alexandru Vasile <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

kad: Revisit FIND_NODE and increase test coverage
2 participants