-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use global_step
while restoring logging step for old checkpoints
#13645
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice catch !
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Co-authored-by: Akihiro Nitta <[email protected]>
Pull request was converted to draft
src/pytorch_lightning/trainer/connectors/checkpoint_connector.py
Outdated
Show resolved
Hide resolved
src/pytorch_lightning/trainer/connectors/checkpoint_connector.py
Outdated
Show resolved
Hide resolved
Codecov Report
@@ Coverage Diff @@
## master #13645 +/- ##
=========================================
- Coverage 86% 76% -10%
=========================================
Files 326 327 +1
Lines 25394 25512 +118
=========================================
- Hits 21839 19446 -2393
- Misses 3555 6066 +2511 |
What does this PR do?
Here #13467, we fixed the issue by restoring the logging step from the checkpoint, but for the cases where old checkpoints (<v1.6.5) are used, the behavior will still be the same because that key will not be present in old checkpoints and logging will start from 0. To avoid this, the default behavior is changed to restore global_step instead.
Discussed with @carmocca
Does your PR introduce any breaking changes? If yes, please list them.
Before submitting
PR review
Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:
Did you have fun?
Make sure you had fun coding 🙃
cc @Borda @carmocca @edward-io @ananthsub @rohitgr7 @kamil-kaczmarek @Raalsky @Blaizzy