[litep2p-milestone]: Improve Kademlia Performance

[lexnv]

The Vision

Enhance the performance and reliability of the litep2p Kademlia implementation to discover nodes faster.

The objective is to close the performance gap between libp2p and litep2p in terms of peer discovery speed. Currently, litep2p is slower (3-4 times on synthetic benchmarks) at discovering peers compared to libp2p.

The Plan

To achieve this, several improvements are required:

• Refactor the Kademlia engine to transition to an event-driven implementation (cc poc: Refactor Kad Engine to poll based implementation #306)
• Retain only reachable addresses in the DHT or utilize the transport manager to extract peer addresses
• Return more than the current limit of 20 closest peers in response to queries
• Provide intermediate KAD record responses, not just upon query completion, to facilitate authority discovery

Tasks

Preview Give feedback

1. kad: Investigate substream reuse and bound the number of open substreams #313
   +0
2. kad: Improve QueryEngine #40
   easy enhancement +2
3. kad: Analyze memory usage #36
   easy enhancement +2
4. Clone only needed KademliaPeers when yielding closest nodes #53
   easy enhancement +2
5. kad: Query::next_action() message is ignored by the kademlia handler #230
   enhancement +1
6. kad: Consider only adding reached addresses to the routing table #209
   enhancement +1
   
7. kad: Add kademlia metrics #171
   enhancement +1
8. kad: Implement query timeouts #168
   enhancement +1
9. kad: Remove QueryAction::QuerySucceeded variant #130
   +0
10. kad: Strategy for removing old MemoryStore records #81
    enhancement +1
11. kad: Implement FindManyNodesContext #80
    enhancement +1
    
12. transport/manager: Refill dial addresses after a dial failure to make dialing more robust #232
    enhancement +1
13. yamux: Update yamux with window tuning and backport fixes #256
    enhancement +1
    
14. Add pending nodes to Kademlia k-buckets #48
    easy enhancement +2
    
15. Implement proper peer eviction #37
    easy enhancement +2
    
16. Increase test coverage for Kademlia #39
    easy +1