Remove Kademlia DHT From Prysm

[nisdas]

💎 Issue

Background

Currently we are running two methods of discovery on Topaz, both kademlia and DiscoveryV5. However running these two methods of discovery can strain the node as it has to maintain two separate dhts. The inital reason we ran kademlia was because our internal infrastructure could support it whereas we did not have a straightforward solution for udp routing in the cluster.

However running kad-dht had brought up some difficult issues for prysm to resolve:
#5663, #5588

In light of the difficulties faced with using kademlia-dht for discovery, it would be better if we could make prysm run only on DiscoveryV5 as this will be the prime method of peer discovery in mainnet.

Description

Tasks Required

• Update our geth fork to the latest version to be up to date with the upstream branch.
• Improve discovery searches to allow us to search by certain enr capabilities
• Have the updated discoveryV5 protocol stable in Topaz before permanently running the network on it.

[prestonvanloon]

Do we have any data on how much resource usage is consumed by kademlia?

[prestonvanloon]

I am wondering if additional discovery could be helpful, maybe we can start with an opt-out feature flag?

[nisdas]

Most of the issues with kademlia are to do in the networking layer, with it continuously dialing private addresses and unroutable peers. Kademlia also does not have a way to throttle active discovery, which can cause it to continously dial new peers. Some users have also reported on their routers being overwhelmed due to large amount of connections constantly being opened from it. cc @terencechain

Since DiscoveryV5 is going to be the default method of discovery in mainnet, we should solely focus on it as that is also what other clients will most likely be only supporting. Running kademlia-dht kind of hides any potential issues with discoveryV5 , as a node could peer with another through kad-dht if discoveryV5 has a bug,etc.

[prestonvanloon]

Sounds good to me. I agree that it doesn't have the flexibility that we need.

[prestonvanloon]

Can we add a FF to disable kad dht? That could be a good way to introduce this change.