cuda crashing in training imagenet

[Cheng-Wang]

My version for caffe is Ubuntu 14.04 LTS with GeForce GTX 780, CUDA 6.0, I found similar issue #39 , and followed the instruction to change the fan speed, but the problem still exists, I hope someone can help me to fix it. Thank you in advance!

I0723 11:11:07.070943 2590 net.cpp:174] Network initialization done.
I0723 11:11:07.070952 2590 net.cpp:175] Memory required for Data 210114408
I0723 11:11:07.070999 2590 solver.cpp:49] Solver scaffolding done.
I0723 11:11:07.071012 2590 solver.cpp:61] Solving CaffeNet
I0723 11:11:07.071033 2590 solver.cpp:106] Iteration 0, Testing net
I0723 11:12:40.819983 2590 solver.cpp:142] Test score #0: 0
I0723 11:12:40.820047 2590 solver.cpp:142] Test score #1: 7.60797
F0723 11:12:41.542250 2590 math_functions.cpp:50] Check failed: status == CUBLAS_STATUS_SUCCESS (14 vs. 0) CUBLAS_STATUS_INTERNAL_ERROR
*** Check failure stack trace: ***

Aborted (core dumped)

[shelhamer]

Sorry, this hardware and software combination has been reported to otherwise work so this is likely some kind of hardware configuration issue.

Please ask any follow-up questions on the caffe-users mailing list. As of the latest release we prefer to keep issues reserved for Caffe development. Thanks!