Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Need help in saving and loading model to resume training #2

Open
BashCache opened this issue Apr 20, 2022 · 0 comments
Open

Need help in saving and loading model to resume training #2

BashCache opened this issue Apr 20, 2022 · 0 comments

Comments

@BashCache
Copy link

We have started to train the model according to the instructions given. We have used CUDA 9.0 , Python 3.6.8, with 1 GPU core. We have trained for 5 ticks and it took nearly 45 minutes for the same. We do not have hardware requirements to keep running the model throughout the day and hence we require your help in knowing the following things:

  1. How long will it take to train the model.
  2. We are assuming network-snapshot pickle file stores our model weights for each tick. We would like to know if our assumption is right and if not, could you please explain what network-snapshot tries to save.
  3. We are planning to run 10 ticks at at time, store the model and resume training from the subsequent tick. We would like to know how to load the model to continue training and would want to know where to make changes in training_loop.py file

It would really be helpful if you could shed light on the above points as soon as possible as this would be immensely helpful for our project. Appreciate your time! Thank you! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant