Resuming Trainer with ckpt_path disrupts logging by TensorBoardLogger

[arsedler9]

🐛 Bug

When I fit a model that uses a TensorBoardLogger and then save a checkpoint with Trainer.save_checkpoint, the TensorBoardLogger of the Trainer that resumes from this checkpoint loses the global step count and is not able to log to the same event files. This causes only the data logged by the second logger to show up in TensorBoard, with the global step count starting at zero.

To Reproduce

Run the CoLab notebook I developed for BoringModel. Note that the logged epochs in TensorBoard go from 10 to 20, starting at step zero.

Expected behavior

I would prefer for the logger to resume logging from the global step count of the Trainer that saved the checkpoint, and for the entire training curves from both trainers to be visible in TensorBoard.

Environment

• CUDA:
  • GPU:
    • Tesla T4
  • available: True
  • version: 11.3
• Packages:
  • numpy: 1.21.6
  • pyTorch_debug: False
  • pyTorch_version: 1.11.0+cu113
  • pytorch-lightning: 1.6.3
  • tqdm: 4.64.0
• System:
  • OS: Linux
  • architecture:
    • 64bit
  • processor: x86_64
  • python: 3.7.13
  • version: # 1 SMP Sun Apr 24 10:03:06 PDT 2022

Additional context

I am using PyTorch Lightning to run Population-Based Training, so I am often saving and resuming models. I'd prefer to have the continuous TensorBoard logs for each worker so I don't end up having to keep track of n_workers * n_generations different traces in TensorBoard.

cc @awaelchli @ananthsub @ninginthecloud @rohitgr7 @carmocca @edward-io

[arsedler9]

BTW, the bug_report_model.ipynb on CoLab is a fantastic idea and well executed! Made it super easy for me to replicate this bug.

[rohitgr7]

I think this is exactly this issue: #12274

[arsedler9]

Agreed! Do you think the issue of overwriting / deleting the TB data from the first training session is related to the step number or could that be a separate issue?

…

On May 10, 2022, at 4:28 AM, Rohit Gupta ***@***.***> wrote:  I think this is exactly this issue: #12274 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

[rohitgr7]

it's different. The logging step used in the previous run is not restored and is instantiating from 0 in the new run.

[rohitgr7]

fixed here: #13467, closing this.