Fix issues with PL 1.8 #5353

SeanNaren · 2022-11-08T15:51:25Z

What does this PR do ?

Attempts to fix all issues after many deprecations/lightning lite becoming the backbone for a lot of the underlying pytorch lightning code.

cc @titu1994

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

Signed-off-by: SeanNaren <[email protected]>

nemo/utils/exp_manager.py

Signed-off-by: SeanNaren <[email protected]>

nemo/collections/tts/models/fastpitch.py

nemo/core/config/hydra_runner.py

nemo/utils/exp_manager.py

Signed-off-by: SeanNaren <[email protected]>

# Conflicts: # nemo/utils/exp_manager.py

Signed-off-by: SeanNaren <[email protected]>

SeanNaren · 2022-11-21T18:46:43Z

This PR should hopefully unblock the move to 1.8: Lightning-AI/pytorch-lightning#15737 (review)

We'll just wait till it gets into a minor release version (which hopefully would be tomorrow).

# Conflicts: # nemo/core/classes/exportable.py

Signed-off-by: SeanNaren <[email protected]>

Jenkinsfile

Signed-off-by: SeanNaren <[email protected]>

titu1994

Overall, incredible work @SeanNaren ! Minor comments and needs one other reviewer since it's a core change

titu1994 · 2022-11-23T23:02:05Z

nemo/core/classes/exportable.py

@@ -128,7 +129,7 @@ def _export(
            # Set module mode
            with torch.onnx.select_model_mode_for_export(
                self, training
-            ), torch.inference_mode(), torch.no_grad(), torch.jit.optimized_execution(True):
+            ), torch.inference_mode(), torch.no_grad(), torch.jit.optimized_execution(True), _jit_is_scripting():


Hmm we don't use lightning during export, is this method safe ?

Would the tests have confirmed if this is fine? The underlying code is simple to what is happening:

In Lightning:

@contextmanager def _jit_is_scripting() -> Generator: """Workaround for https://github.com/pytorch/pytorch/issues/67146.""" LightningModule._jit_is_scripting = True try: yield finally: LightningModule._jit_is_scripting = False

Hmm i dont think it would be caught in tests, lets keep it for now

tests/hydra/test_hydra_runner.py

Signed-off-by: SeanNaren <[email protected]>

lgtm-com · 2022-11-24T10:43:27Z

This pull request fixes 1 alert when merging d614378 into 2430d0f - view on LGTM.com

fixed alerts:

1 for Unused import

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog.

* Fix issues with PL 1.8 Signed-off-by: SeanNaren <[email protected]> * Set scripting variable Signed-off-by: SeanNaren <[email protected]> * Fix missing arg Signed-off-by: SeanNaren <[email protected]> * Cleanup list Signed-off-by: SeanNaren <[email protected]> * Fix reference Signed-off-by: SeanNaren <[email protected]> * Try to fix hanging EMA test Signed-off-by: SeanNaren <[email protected]> * Missing \ Signed-off-by: SeanNaren <[email protected]> * Add strategy Signed-off-by: SeanNaren <[email protected]> * See if setting the chdir fixes the hanging DDP test Signed-off-by: SeanNaren <[email protected]> * See if removing the subdir setter fixes the issue Signed-off-by: SeanNaren <[email protected]> * Remove checks Signed-off-by: SeanNaren <[email protected]> * Try [0,1] for devices Signed-off-by: SeanNaren <[email protected]> * Add code back Signed-off-by: SeanNaren <[email protected]> * Remove space Signed-off-by: SeanNaren <[email protected]> * Update requirements Signed-off-by: SeanNaren <[email protected]> * Swap import path Signed-off-by: SeanNaren <[email protected]> * Update references Signed-off-by: SeanNaren <[email protected]> * Fix deprecated variables Signed-off-by: SeanNaren <[email protected]> * Fix missing var Signed-off-by: SeanNaren <[email protected]> * Fix var Signed-off-by: SeanNaren <[email protected]> * Revert changes Signed-off-by: SeanNaren <[email protected]> * Address review Signed-off-by: SeanNaren <[email protected]> Signed-off-by: SeanNaren <[email protected]> Co-authored-by: Oleksii Kuchaiev <[email protected]> Signed-off-by: Hainan Xu <[email protected]>

* Fix issues with PL 1.8 Signed-off-by: SeanNaren <[email protected]> * Set scripting variable Signed-off-by: SeanNaren <[email protected]> * Fix missing arg Signed-off-by: SeanNaren <[email protected]> * Cleanup list Signed-off-by: SeanNaren <[email protected]> * Fix reference Signed-off-by: SeanNaren <[email protected]> * Try to fix hanging EMA test Signed-off-by: SeanNaren <[email protected]> * Missing \ Signed-off-by: SeanNaren <[email protected]> * Add strategy Signed-off-by: SeanNaren <[email protected]> * See if setting the chdir fixes the hanging DDP test Signed-off-by: SeanNaren <[email protected]> * See if removing the subdir setter fixes the issue Signed-off-by: SeanNaren <[email protected]> * Remove checks Signed-off-by: SeanNaren <[email protected]> * Try [0,1] for devices Signed-off-by: SeanNaren <[email protected]> * Add code back Signed-off-by: SeanNaren <[email protected]> * Remove space Signed-off-by: SeanNaren <[email protected]> * Update requirements Signed-off-by: SeanNaren <[email protected]> * Swap import path Signed-off-by: SeanNaren <[email protected]> * Update references Signed-off-by: SeanNaren <[email protected]> * Fix deprecated variables Signed-off-by: SeanNaren <[email protected]> * Fix missing var Signed-off-by: SeanNaren <[email protected]> * Fix var Signed-off-by: SeanNaren <[email protected]> * Revert changes Signed-off-by: SeanNaren <[email protected]> * Address review Signed-off-by: SeanNaren <[email protected]> Signed-off-by: SeanNaren <[email protected]> Co-authored-by: Oleksii Kuchaiev <[email protected]> Signed-off-by: andrusenkoau <[email protected]>

Fix issues with PL 1.8

477baaa

Signed-off-by: SeanNaren <[email protected]>

SeanNaren mentioned this pull request Nov 8, 2022

Move to optimizer based EMA implementation #5169

Merged

8 tasks

SeanNaren commented Nov 8, 2022

View reviewed changes

nemo/utils/exp_manager.py Show resolved Hide resolved

SeanNaren added 9 commits November 8, 2022 16:19

Set scripting variable

c6eb78c

Signed-off-by: SeanNaren <[email protected]>

Fix missing arg

4cc6301

Signed-off-by: SeanNaren <[email protected]>

Cleanup list

25e0f77

Signed-off-by: SeanNaren <[email protected]>

Fix reference

60db72b

Signed-off-by: SeanNaren <[email protected]>

Merge branch 'main' into feat/lightning_1.8_support

0ed8d1e

Try to fix hanging EMA test

6ef98e7

Signed-off-by: SeanNaren <[email protected]>

Missing \

e25704c

Signed-off-by: SeanNaren <[email protected]>

Add strategy

4da05eb

Signed-off-by: SeanNaren <[email protected]>

See if setting the chdir fixes the hanging DDP test

4e6913c

Signed-off-by: SeanNaren <[email protected]>

titu1994 reviewed Nov 9, 2022

View reviewed changes

nemo/collections/tts/models/fastpitch.py Show resolved Hide resolved

nemo/core/config/hydra_runner.py Outdated Show resolved Hide resolved

nemo/utils/exp_manager.py Show resolved Hide resolved

SeanNaren and others added 7 commits November 10, 2022 11:49

See if removing the subdir setter fixes the issue

07d4292

Signed-off-by: SeanNaren <[email protected]>

Merge branch 'main' into feat/lightning_1.8_support

9c6307a

# Conflicts: # nemo/utils/exp_manager.py

Remove checks

15087e5

Signed-off-by: SeanNaren <[email protected]>

Try [0,1] for devices

4a01cec

Signed-off-by: SeanNaren <[email protected]>

Merge branch 'main' into feat/lightning_1.8_support

6c3e417

Merge branch 'main' into feat/lightning_1.8_support

1aad730

Add code back

b1add40

Signed-off-by: SeanNaren <[email protected]>

SeanNaren mentioned this pull request Nov 15, 2022

Hydra + DDP Fails for NeMo after Hydra refactor in 1.8 Lightning-AI/pytorch-lightning#15689

Closed

Remove space

8c7d7e0

Signed-off-by: SeanNaren <[email protected]>

SeanNaren added 7 commits November 23, 2022 11:10

Merge branch 'main' into feat/lightning_1.8_support

dce4918

# Conflicts: # nemo/core/classes/exportable.py

Update requirements

9e8ab6b

Signed-off-by: SeanNaren <[email protected]>

Swap import path

b33501d

Signed-off-by: SeanNaren <[email protected]>

Update references

580654c

Signed-off-by: SeanNaren <[email protected]>

Fix deprecated variables

ed8df6f

Signed-off-by: SeanNaren <[email protected]>

Fix missing var

12d7ea2

Signed-off-by: SeanNaren <[email protected]>

Fix var

0e8e59a

Signed-off-by: SeanNaren <[email protected]>

Merge branch 'main' into feat/lightning_1.8_support

8efe28f

SeanNaren commented Nov 23, 2022

View reviewed changes

Jenkinsfile Outdated Show resolved Hide resolved

Revert changes

d672304

Signed-off-by: SeanNaren <[email protected]>

SeanNaren marked this pull request as ready for review November 23, 2022 21:00

Merge branch 'main' into feat/lightning_1.8_support

706b19c

titu1994 reviewed Nov 23, 2022

View reviewed changes

Address review

d614378

Signed-off-by: SeanNaren <[email protected]>

titu1994 merged commit d349f4a into main Nov 28, 2022

titu1994 deleted the feat/lightning_1.8_support branch November 28, 2022 21:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix issues with PL 1.8 #5353

Fix issues with PL 1.8 #5353

SeanNaren commented Nov 8, 2022

SeanNaren commented Nov 21, 2022

titu1994 left a comment

titu1994 Nov 23, 2022

SeanNaren Nov 24, 2022

titu1994 Nov 28, 2022

lgtm-com bot commented Nov 24, 2022

Fix issues with PL 1.8 #5353

Fix issues with PL 1.8 #5353

Conversation

SeanNaren commented Nov 8, 2022

What does this PR do ?

Before your PR is "Ready for review"

SeanNaren commented Nov 21, 2022

titu1994 left a comment

Choose a reason for hiding this comment

titu1994 Nov 23, 2022

Choose a reason for hiding this comment

SeanNaren Nov 24, 2022

Choose a reason for hiding this comment

titu1994 Nov 28, 2022

Choose a reason for hiding this comment

lgtm-com bot commented Nov 24, 2022