Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use logging in benchmarks, fix libp2p-related issues #280

Merged
merged 11 commits into from
Jun 17, 2021
Merged

Use logging in benchmarks, fix libp2p-related issues #280

merged 11 commits into from
Jun 17, 2021

Conversation

justheuristic
Copy link
Member

@justheuristic justheuristic commented Jun 17, 2021 •
edited
Loading

  • rewrite benchmark_dht.py to use logging
  • rewrite benchmark_throughput.py to use logging
  • rewrite benchmark_averaging.py to use logging
  • rewrite benchmark_tensor_comrpession.py to use logging
  • fixed a minor bug that randomly printed EOFError at the endwhen terminating DecentralizedAverager
  • added minor readability tweak to setup.py
  • fix flaky p2p test
  • fix benchmark_throughput hanging
  • add GHA for benchmarks

New benchmark_dht output:

[2021/06/17 12:59:44.354][INFO][root.benchmark_dht:23] Creating peers...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [00:02<00:00, 12.36it/s]
[2021/06/17 12:59:46.946][INFO][root.benchmark_dht:35] Sampled 256 unique ids (after deduplication)
[2021/06/17 12:59:46.947][INFO][root.benchmark_dht:38] Storing experts to dht in batches of 32...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:03<00:00,  2.25it/s]
[2021/06/17 12:59:50.506][INFO][root.benchmark_dht:55] Store success rate: 100.0% (768 / 768)
[2021/06/17 12:59:50.506][INFO][root.benchmark_dht:56] Mean store time: 0.0046285, Total: 3.5547
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 20.26it/s]
[2021/06/17 12:59:50.902][INFO][root.benchmark_dht:77] Get success rate: 100.0 (256 / 256)
[2021/06/17 12:59:50.903][INFO][root.benchmark_dht:78] Mean get time: 0.00153, Total: 0.39125
[2021/06/17 12:59:50.904][INFO][root.benchmark_dht:81] Node survival rate: 100.000%

New benchmark_averaging.py output:

(hivemind_py39) jheuristic@nora:/storage/hdd1/jheuristic/exp/decentralized/jheuristic/hivemind_20210617/benchmarks$ python benchmark_averaging.py --target_group_size 4
[2021/06/17 12:53:42.586][INFO][root.run_averager:59] Averager 0: started on endpoint ipv4:127.0.0.1:36397, group_bits: 00
[2021/06/17 12:53:42.685][INFO][root.run_averager:59] Averager 1: started on endpoint ipv4:127.0.0.1:39463, group_bits: 01
[2021/06/17 12:53:42.757][INFO][root.run_averager:59] Averager 2: started on endpoint ipv4:127.0.0.1:43499, group_bits: 10
...
[2021/06/17 12:53:44.003][INFO][root.run_averager:59] Averager 13: started on endpoint ipv4:127.0.0.1:38575, group_bits: 01
[2021/06/17 12:53:44.264][INFO][root.run_averager:59] Averager 14: started on endpoint ipv4:127.0.0.1:34481, group_bits: 10
[2021/06/17 12:53:44.418][INFO][root.run_averager:59] Averager 15: started on endpoint ipv4:127.0.0.1:43771, group_bits: 11
[2021/06/17 12:53:51.883][INFO][root.run_averager:68] Averager 12: finished step 0
[2021/06/17 12:53:52.575][INFO][root.run_averager:68] Averager 8: finished step 0
[2021/06/17 12:53:52.587][INFO][root.run_averager:68] Averager 4: finished step 0
[2021/06/17 12:53:52.745][INFO][root.run_averager:68] Averager 6: finished step 0
...
[2021/06/17 12:54:10.757][INFO][root.run_averager:68] Averager 12: finished step 4
[2021/06/17 12:54:10.757][INFO][root.run_averager:69] Averager 12: done.
[2021/06/17 12:54:10.758][INFO][root.run_averager:68] Averager 15: finished step 3
[2021/06/17 12:54:25.764][INFO][root.run_averager:68] Averager 15: failed step 4	
[2021/06/17 12:54:25.764][INFO][root.run_averager:69] Averager 15: done.
[2021/06/17 12:54:25.765][INFO][root.benchmark_averaging:82] Benchmark finished in 41.586 seconds.
[2021/06/17 12:54:25.765][INFO][root.benchmark_averaging:83] Success rate: 0.9875 (79 out of 80 attempts)

@justheuristic justheuristic requested review from borzunov, mryab and yhn112 and removed request for borzunov June 17, 2021 12:58
@justheuristic justheuristic marked this pull request as draft June 17, 2021 13:01
@codecov
Copy link

codecov bot commented Jun 17, 2021 •
edited
Loading

Codecov Report

Merging #280 (0ea49aa) into master (40277b6) will decrease coverage by 0.01%.
The diff coverage is 50.00%.

@@            Coverage Diff             @@
##           master     #280      +/-   ##
==========================================
- Coverage   72.91%   72.89%   -0.02%     
==========================================
  Files          62       62              
  Lines        5589     5594       +5     
==========================================
+ Hits         4075     4078       +3     
- Misses       1514     1516       +2
Impacted Files Coverage Δ
hivemind/client/averaging/__init__.py 48.36% <40.00%> (-0.29%) ⬇️
hivemind/server/runtime.py 75.00% <100.00%> (+1.12%) ⬆️
hivemind/utils/timed_storage.py 92.85% <0.00%> (-1.03%) ⬇️
hivemind/dht/protocol.py 92.53% <0.00%> (-0.83%) ⬇️
hivemind/dht/__init__.py 72.51% <0.00%> (ø)
hivemind/server/__init__.py 88.53% <0.00%> (+0.63%) ⬆️
hivemind/server/dht_handler.py 58.82% <0.00%> (+1.96%) ⬆️
hivemind/server/layers/dropout.py 96.87% <0.00%> (+3.12%) ⬆️

@justheuristic justheuristic marked this pull request as ready for review June 17, 2021 15:51
@mryab mryab changed the title update benchmarks to use logging Update benchmarks to use logging Jun 17, 2021
@mryab mryab changed the title Update benchmarks to use logging Use logging in benchmarks, fix libp2p-related issues Jun 17, 2021
@justheuristic justheuristic merged commit 42b9b6c into master Jun 17, 2021
@justheuristic justheuristic deleted the fix_p2p_flakiness branch June 17, 2021 19:35
@justheuristic justheuristic self-assigned this Jun 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mryab mryab mryab approved these changes

@yhn112 yhn112 Awaiting requested review from yhn112

Assignees

@justheuristic justheuristic

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@justheuristic @mryab