restore epoch and step information when resuming training

[thewh1teagle]

When resuming training from checkpoint it starts from 0 epoch and 0 step although it should be much higher and I can hear that it's already trained.
Can you fix it so it will restore it so I can keep track on the step counter when resuming?

Lightning-AI/pytorch-lightning#12274

[rmcpantoja]

hi,
Is this error occurring in --restore-from-checkpoint argument from the trainer? That didn't happen when I used it, although I think it is used more for finetuning purposes. If not, can you try that argument?
Cheers.

[thewh1teagle]

occurring in --restore-from-checkpoint argument from the trainer?

Hey,
I don't see this argument in the trainer, maybe your'e talking about old version?

https://github.com/mush42/optispeech/blob/main/configs/train.yaml

[mush42]

@thewh1teagle
are you using the forced_resume argument?
I added this to be able to (re)load the model when I make changes to it's architecture while benefiting from already trained layers.
It is very niche use case, but I added it to save myself some time during development.

If you want to resume training normally, then the ckpt_path is all you need.