-
Notifications
You must be signed in to change notification settings - Fork 69
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The Loss was always 87.3365. Is that normal or there was some errors in it. Thanks #2
Comments
Definitely you have an issue with your install. It should look something like the log below. You can check each install step, and particularly that your PYTHONPATH doesn't have an other Caffe versions or something. I1222 13:13:32.939466 3413 solver.cpp:218] Iteration 0 (-2.29907e-05 iter/s, 1.32516s/100 iters), loss = 184.287 |
It seems that if I use 4 gpus for training, the loss always be 87.3, |
To change from 2 gpus to 4 gpus, I think there are changes that would need to be made in several parts of the code. It would be best to build a new experiment using create_caption_lstm.py, changing |
I0502 17:46:41.519320 22378 solver.cpp:218] Iteration 0 (0 iter/s, 1.67459s/100 iters), loss = 184.354
I0502 17:46:41.519423 22378 solver.cpp:237] Train net output #0: accuracy = 0
I0502 17:46:41.519461 22378 solver.cpp:237] Train net output #1: cross_entropy_loss = 9.21768 (* 20 = 184.354 loss)
I0502 17:46:41.519623 22378 sgd_solver.cpp:105] Iteration 0, lr = 0.01
I0502 17:48:26.913466 22378 solver.cpp:218] Iteration 100 (0.948815 iter/s, 105.395s/100 iters), loss = 1746.73
I0502 17:48:26.913585 22378 solver.cpp:237] Train net output #0: accuracy = 1
I0502 17:48:26.913609 22378 solver.cpp:237] Train net output #1: cross_entropy_loss = 87.3365 (* 20 = 1746.73 loss)
I0502 17:48:26.913686 22378 sgd_solver.cpp:105] Iteration 100, lr = 0.00998333
I0502 17:50:12.863924 22378 solver.cpp:218] Iteration 200 (0.943834 iter/s, 105.951s/100 iters), loss = 1746.73
I0502 17:50:12.864044 22378 solver.cpp:237] Train net output #0: accuracy = 1
I0502 17:50:12.864061 22378 solver.cpp:237] Train net output #1: cross_entropy_loss = 87.3365 (* 20 = 1746.73 loss)
I0502 17:50:12.864110 22378 sgd_solver.cpp:105] Iteration 200, lr = 0.00996667
I0502 17:51:58.913350 22378 solver.cpp:218] Iteration 300 (0.942953 iter/s, 106.05s/100 iters), loss = 1746.73
I0502 17:51:58.913466 22378 solver.cpp:237] Train net output #0: accuracy = 1
I0502 17:51:58.913488 22378 solver.cpp:237] Train net output #1: cross_entropy_loss = 87.3365 (* 20 = 1746.73 loss)
The text was updated successfully, but these errors were encountered: