Improve Optimizer docs, update quickstart to use Optimizer #416
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
justheuristic
commented
Nov 29, 2021
justheuristic
commented
- deduplicate docs, fix typos, improve formatting
- switch quickstart.md to the new optimizer
Codecov Report
@@ Coverage Diff @@
## master #416 +/- ##
==========================================
+ Coverage 83.57% 83.67% +0.09%
==========================================
Files 77 77
Lines 7788 7790 +2
==========================================
+ Hits 6509 6518 +9
+ Misses 1279 1272 -7
|
justheuristic
changed the title
borzunov
requested changes
Nov 30, 2021
docs/modules/optim.rst
Outdated
| @@ -26,9 +29,9 @@ | |||
|
|
|||
| .. raw:: html | |||
|
|
|||
| CollaborativeOptimizer is a legacy version of hivemind.Optimizer. **For new projects, please use hivemind.Optimizer.** | |||
| CollaborativeOptimizer is a legacy version of hivemind.Optimizer. <b>For new projects please use hivemind.Optimizer</b>. | |||
| Currently, hivemind.Optimizer supports all the features of CollaborativeOptimizer and then some. | |||
There was a problem hiding this comment.
supports all the features of CollaborativeOptimizer and then some.
The sentence seems to be not finished.
There was a problem hiding this comment.
I respectfully disagree [1]
[1] - https://www.urbandictionary.com/define.php?term=and%20then%20some
There was a problem hiding this comment.
I think this phrase is not widely-known enough to be used in the docs. Most readers just won't understand it :(
Co-authored-by: Alexander Borzunov <[email protected]>
…mizer-docs-review
borzunov
reviewed
Dec 2, 2021
| >>> # alternative: opt = hivemind.Optimizer(dht, run_id="run_42", optimizer=torch.optim.Adam(model.parameters()) | ||
| >>> opt = hivemind.Optimizer(dht=dht, run_id="run_42", params=model.parameters(), | ||
| >>> optimizer=lambda params: torch.optim.Adam(params, **other_options), | ||
| >>> target_batch_size=4096, batch_size_per_step=4) |
There was a problem hiding this comment.
Let's use the same order of parameters in hivemind.Optimizer definition, both examples in the API docs and the quickstart. It makes it easier to compare the values of the same arguments in these examples, as well as go through them to check whether you've specified all you need in your own code.
borzunov
reviewed
Dec 2, 2021
| ``` | ||
|
|
||
| Congrats, you've just started a pocket-sized experiment with decentralized deep learning! | ||
|
|
||
| However, this is just the bare minimum of what hivemind can do. In [this example](https://github.com/learning-at-home/hivemind/tree/master/examples/albert), | ||
| However, this is only the basics of what hivemind can do. In [this example](https://github.com/learning-at-home/hivemind/tree/master/examples/albert), | ||
| we show how to use a more advanced version of DecentralizedOptimizer to collaboratively train a large Transformer over the internet. |
There was a problem hiding this comment.
Please replace DecentralizedOptimizer -> hivemind.Optimizer here and in L186.
borzunov
reviewed
Dec 2, 2021
| ``` | ||
|
|
||
| Congrats, you've just started a pocket-sized experiment with decentralized deep learning! | ||
|
|
||
| However, this is just the bare minimum of what hivemind can do. In [this example](https://github.com/learning-at-home/hivemind/tree/master/examples/albert), | ||
| However, this is only the basics of what hivemind can do. In [this example](https://github.com/learning-at-home/hivemind/tree/master/examples/albert), | ||
| we show how to use a more advanced version of DecentralizedOptimizer to collaboratively train a large Transformer over the internet. | ||
|
|
||
| If you want to learn more about each individual component, |
There was a problem hiding this comment.
Please replace:
(Li et al. 2020)->Li et al. (2020)(Ryabinin et al. 2021)->Ryabinin et al. (2021)
borzunov
approved these changes
Dec 2, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.