Skip to content

Commit

Permalink
update docs
Browse files Browse the repository at this point in the history
  • Loading branch information
rohitgr7 committed Aug 26, 2022
1 parent 42db8a0 commit 84adb6e
Showing 1 changed file with 3 additions and 2 deletions.
5 changes: 3 additions & 2 deletions docs/source-pytorch/advanced/model_parallel.rst
Original file line number Diff line number Diff line change
Expand Up @@ -250,7 +250,8 @@ Lightning supports. The API is pretty similar to that of FairScale.


Auto Wrapping
"""""""""""""
=============

Model layers should be wrapped in FSDP in a nested way to save peak memory and enable communication and computation overlapping. The
simplest way to do it is auto wrapping, which can serve as a drop-in replacement for DDP without changing the rest of the code. You don't
have to ``wrap`` layers manually as in the case of manual wrapping.
Expand All @@ -266,7 +267,7 @@ Read more `here <https://pytorch.org/blog/introducing-pytorch-fully-sharded-data


Manual Wrapping
"""""""""""""""
===============

Manual wrapping can be useful to explore complex sharding strategies by applying ``wrap`` selectively to some parts of the model. To activate
parameter sharding with manual wrapping, you can wrap your model using the ``wrap`` function. Internally in Lightning, we enable a context manager around the ``configure_sharded_model`` function to make sure the ``wrap`` parameters are passed correctly.
Expand Down

0 comments on commit 84adb6e

Please sign in to comment.