Add ability to persist and load peers to/from JSON

[RangerMauve]

As discussed on IRC, Russia is going to play around with closing it's network off from the rest of the internet.

This could kill Dat applications by blocking the DHT bootstrap nodes.

A good step would be to support persisting known nodes to help bootstrapping, and to account for scenarios where the bootstrap nodes aren't accessible.

[da2x]

Context: Distributed web not ready for Runet cutoff from the Internet

BitTorrent clients solve this by dumping all the working DHT nodes it has connected to a file. On startup, the client tries to connect to previously seen DHT nodes and only uses the centralized bootstrappers as a fallback if none of the previous DHT nodes are still reachable. This introduces diversity in the network and reduces its dependency on centralized [blockable] infrastructure.

This approach also reduces load on the bootstrapper servers as only new clients will need to connect to them. It also slightly improves client security as they don't reach out to domains that may be monitored more often than necessary.

I'm not aware if BitTorrent clients try to rank which DHT nodes are most stable and prioritize connections, or if they just begin randomly reaching out to all the previously seen nodes.

[RangerMauve]

The bittorrent-dht module has a toJSON method that dumps the state of the known nodes and values. Maybe the same thing could be adopted?

Would a PR be welcome?

[mafintosh]

PR very welcome

…

On Thu, 14 Feb 2019 at 17.52, RangerMauve ***@***.***> wrote: The bittorrent-dht module has a toJSON <https://www.npmjs.com/package/bittorrent-dht#arr--dhttojson> method that dumps the state of the known nodes and values. Maybe the same thing could be adopted? Would a PR be welcome? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAW_VYQgZhKl-O0OhFk_Sr8akttJzOgPks5vNZRjgaJpZM4a7_L2> .

[martinheidegger]

I am wondering if a level-db would be more suitable for this?

[make-github-pseudonymous-again]

Does this need to be implemented in this DHT or in the DHT in dht-rpc which this DHT extends?

[mafintosh]

@aureooms dht-rpc needs a way to import an array of nodes and export. the export you can already do with dht.nodes.toArray(), unsure if we have an api for import, but that shouldn't be too hard.

[make-github-pseudonymous-again]

Should import functionality only add input nodes as bootstrap nodes? In that case the API almost already exists, just construct a Node with the appropriate bootstrap option. Maybe needs special handling for default bootstrap nodes, avoid querying them if there is no need to. Then export only needs to store host and port.

We could also have a more complex logic where we store other properties (ids, tokens, and whatnot) and import all of that. Then directly import them through calls to _addNode?

[mafintosh]

True you can use the bootstrap option, but option 2 is better, as it can then fall back on the bootstrappers if all nodes are failing. the dht.nodes.toArray() gives you all nodes with the id etc.

[make-github-pseudonymous-again]

I opened holepunchto/dht-rpc#21 to continue this discussion if this needs to be implemented in dht-rpc.

[mafintosh]

addNodes, getNodes landed in latest dht-rpc which this inherits from.

[mafintosh]

For persistence that's something we should discuss in hyperdrive-daemon (in the hyper case)