v2 colab: Monitoring runs when no logs or tensorboard working?

[drscotthawley]

Hi, I'm using @hexorcismos' RAVE v2 training notebook. I created a directory called /content/logs and specified that as my logs directory.

But during training, nothing ever seems to get written to that directory. Consequently tensorboard never shows anything.

How are we to monitor our runs, and/or decide when to stop training? Thanks.

[drscotthawley]

Here's a picture of the /content/ directory during training. You can see how "logs" remains empty.

It seems that TF events are instead being written inside the checkpoints directory instead of the logs direcotory, but there's no way to point Tensorboard to this new directory once it's started.

[drscotthawley]

Ok, so @hexorcismos answered my tweet question, saying

Im about to board a flight, but quick answer is: run your rave training commando from the directory you want to save to. It makes a “runs” folder in that same path…

Which means you can't specify any directory for logs_path. Instead, it must match your save_dir + /runs or else you'll never see anything.

So, I'm going to modify the notebook so that you specify save_dir earlier before the tensorboard part, and remove the "log_path" prompt completely.

Will post a link to my changed notebook after I verify that it works.
Update: here you go: https://colab.research.google.com/drive/1-w6x9smtfSDk6sQ_nEocUF6OHU6qkkAf?usp=sharing

This has a couple lines specific to my dataset, but more generally:

• I added a "-y" to the conda call so it wouln't stop to ask
• made sure it creates logging the directory it needs
• made sure that tensorboard is looking in the correct directory.

[iamisobe]

are there any tutorials that visually show how to run the notebook? im not really understanding entirely how to start the training

[ariannanellospazio]

I'd like to thank you so much, with your explanation the code now works! I spent the whole day figuring out where the problem could be

[drscotthawley]

...also wondering when to expect the first checkpoint to be written? I'm now at 200 epochs and still no checkpoints.

Edit: Ok, apparently the default is every 10k steps. Now that I've passed that threshold, I see a checkpoint file.