Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert DHT to libp2p backend #296

Merged
merged 86 commits into from
Jul 10, 2021
Merged

Convert DHT to libp2p backend #296

merged 86 commits into from
Jul 10, 2021

Conversation

borzunov
Copy link
Member

@borzunov borzunov commented Jun 29, 2021

Current status: finishing the docs and minor details, ready for review.

What I have tested:

  • Unit tests work
  • Benchmarks work
  • In examples/albert, both run_training_monitor.py and run_trainer.py work. Training with 2 GPU peers + 1 training monitor performs successful averagings and gradually reduces the loss.

Plan:

  • Make DHTProtocol work over P2P
  • Make DHTNode work over P2P
  • Make hivemind.dht.DHT work over P2P
  • Convert test_dht_node.py
  • Convert test_dht.py
  • Convert other tests matching test_dht_*.py
  • Convert tests not related to DHT directly
  • Convert DHT usages outside tests
  • Fix docs (initial_peers/listen_on/dht_port/port in *.md)
  • Fix duplicated logging in run_training_monitor.py

Details:

  • Fix AuthRPCWrapper usage with Role = Server
  • Terminate all p2pd instances
  • Remove all Unix socket connections (and not interfere with other processes)
  • Handle SIGTERM in test_dht_node.py
  • Mimick gRPC servicer (make ServicerBase and a P2P method to add it?)
  • Other TODOs (if present)

Ideas for future PRs:

  • Add test for use_ipfs=True
  • Implement simplified multiaddr sharing
  • Thoroughly check graceful shutdown
  • Call drain() after all StreamWriter.writes
  • Rewrite do_handle_stream(): send data in chunks, call drain(), don't concat a big string with its size

Related issues: #91 #149 #165 #185 #238

@borzunov borzunov added the dht label Jun 29, 2021
@borzunov borzunov force-pushed the dht-over-p2p branch 2 times, most recently from 8c567eb to 3c9e859 Compare July 10, 2021 05:13
@borzunov borzunov requested a review from mryab July 10, 2021 06:57
tests/test_dht_node.py Outdated Show resolved Hide resolved
tests/test_utils/dht_swarms.py Outdated Show resolved Hide resolved
@borzunov borzunov changed the title Make DHT work over libp2p Convert DHT to libp2p backend Jul 10, 2021
@borzunov borzunov merged commit 0be1512 into master Jul 10, 2021
@borzunov borzunov deleted the dht-over-p2p branch July 10, 2021 14:54
borzunov added a commit that referenced this pull request Jul 11, 2021
This PR follows #296, removes importing PeerID as Endpoint in dht/{node,protocol,routing}.py and related tests, and performs a number of replacements like Endpoint -> PeerID and endpoint -> peer_id.
@borzunov borzunov mentioned this pull request Jul 11, 2021
31 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants