Check failed: error == cudaSuccess (11 vs. 0) invalid argument

[Elasine]

I0411 17:13:50.459023 7212 net.cpp:226] conv5_2 needs backward computation.
I0411 17:13:50.459028 7212 net.cpp:226] relu5_1 needs backward computation.
I0411 17:13:50.459031 7212 net.cpp:226] conv5_1 needs backward computation.
I0411 17:13:50.459034 7212 net.cpp:226] pool4 needs backward computation.
I0411 17:13:50.459036 7212 net.cpp:226] relu4_3 needs backward computation.
I0411 17:13:50.459038 7212 net.cpp:226] conv4_3 needs backward computation.
I0411 17:13:50.459040 7212 net.cpp:226] relu4_2 needs backward computation.
I0411 17:13:50.459043 7212 net.cpp:226] conv4_2 needs backward computation.
I0411 17:13:50.459045 7212 net.cpp:226] relu4_1 needs backward computation.
I0411 17:13:50.459048 7212 net.cpp:226] conv4_1 needs backward computation.
I0411 17:13:50.459050 7212 net.cpp:226] pool3 needs backward computation.
I0411 17:13:50.459053 7212 net.cpp:226] relu3_3 needs backward computation.
I0411 17:13:50.459055 7212 net.cpp:226] conv3_3 needs backward computation.
I0411 17:13:50.459058 7212 net.cpp:226] relu3_2 needs backward computation.
I0411 17:13:50.459059 7212 net.cpp:226] conv3_2 needs backward computation.
I0411 17:13:50.459062 7212 net.cpp:226] relu3_1 needs backward computation.
I0411 17:13:50.459064 7212 net.cpp:226] conv3_1 needs backward computation.
I0411 17:13:50.459066 7212 net.cpp:228] pool2 does not need backward computation.
I0411 17:13:50.459070 7212 net.cpp:228] relu2_2 does not need backward computation.
I0411 17:13:50.459072 7212 net.cpp:228] conv2_2 does not need backward computation.
I0411 17:13:50.459074 7212 net.cpp:228] relu2_1 does not need backward computation.
I0411 17:13:50.459077 7212 net.cpp:228] conv2_1 does not need backward computation.
I0411 17:13:50.459079 7212 net.cpp:228] pool1 does not need backward computation.
I0411 17:13:50.459081 7212 net.cpp:228] relu1_2 does not need backward computation.
I0411 17:13:50.459085 7212 net.cpp:228] conv1_2 does not need backward computation.
I0411 17:13:50.459087 7212 net.cpp:228] relu1_1 does not need backward computation.
I0411 17:13:50.459089 7212 net.cpp:228] conv1_1 does not need backward computation.
I0411 17:13:50.459092 7212 net.cpp:228] gt_boxes_input-data_2_split does not need backward computation.
I0411 17:13:50.459095 7212 net.cpp:228] im_info_input-data_1_split does not need backward computation.
I0411 17:13:50.459098 7212 net.cpp:228] data_input-data_0_split does not need backward computation.
I0411 17:13:50.459102 7212 net.cpp:228] input-data does not need backward computation.
I0411 17:13:50.459105 7212 net.cpp:270] This network produces output loss_bbox
I0411 17:13:50.459106 7212 net.cpp:270] This network produces output loss_cls
I0411 17:13:50.459110 7212 net.cpp:270] This network produces output rpn_cls_loss
I0411 17:13:50.459112 7212 net.cpp:270] This network produces output rpn_loss_bbox
I0411 17:13:50.459147 7212 net.cpp:283] Network initialization done.
I0411 17:13:50.459319 7212 solver.cpp:60] Solver scaffolding done.
Loading pretrained model weights from data/imagenet_models/VGG16.v2.caffemodel
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:505] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 553432430
I0411 17:13:51.159070 7212 net.cpp:816] Ignoring source layer pool5
I0411 17:13:51.266799 7212 net.cpp:816] Ignoring source layer fc8
I0411 17:13:51.266815 7212 net.cpp:816] Ignoring source layer prob
Solving...
/home/elasine/py-faster-rcnn/tools/../lib/rpn/proposal_target_layer.py:167: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
fg_inds = npr.choice(fg_inds, size=fg_rois_per_this_image, replace=False)
/home/elasine/py-faster-rcnn/tools/../lib/rpn/proposal_target_layer.py:178: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
bg_inds = npr.choice(bg_inds, size=bg_rois_per_this_image, replace=False)
/home/elasine/py-faster-rcnn/tools/../lib/rpn/proposal_target_layer.py:185: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
labels[fg_rois_per_this_image:] = 0
I0411 17:13:52.418354 7212 solver.cpp:229] Iteration 0, loss = 5.59988
I0411 17:13:52.418401 7212 solver.cpp:245] Train net output #0: loss_bbox = 0.683362 (* 1 = 0.683362 loss)
I0411 17:13:52.418406 7212 solver.cpp:245] Train net output #1: loss_cls = 3.94675 (* 1 = 3.94675 loss)
I0411 17:13:52.418411 7212 solver.cpp:245] Train net output #2: rpn_cls_loss = 0.891574 (* 1 = 0.891574 loss)
I0411 17:13:52.418414 7212 solver.cpp:245] Train net output #3: rpn_loss_bbox = 0.0216602 (* 1 = 0.0216602 loss)
I0411 17:13:52.418419 7212 sgd_solver.cpp:106] Iteration 0, lr = 0.001
F0411 17:13:52.421104 7212 sgd_solver.cu:19] Check failed: error == cudaSuccess (11 vs. 0) invalid argument
*** Check failure stack trace: ***
./experiments/scripts/faster_rcnn_end2end.sh: 行 57: 7212 aborted core dumped ./tools/train_net.py --gpu ${GPU_ID} --solver models/${PT_DIR}/${NET}/faster_rcnn_end2end/solver.prototxt --weights data/imagenet_models/${NET}.v2.caffemodel --imdb ${TRAIN_IMDB} --iters ${ITERS} --cfg experiments/cfgs/faster_rcnn_end2end.yml ${EXTRA_ARGS}

The mistake is above.Some information of mine:Ubuntu16.04,Cuda8.0.44,Cudnn5.I'd appreciate it if you can help me solve it.

[surajitsaikia27]

The only problem could be you don't have enough GPU memory or another program is occupying the GPU. Before running just check how much GPU resource you have by typing nvidia-smi.

[Elasine]

@surajitsaikia27 Thank you all the same.When I change compute capability of GPU according to my computer, it is normal.

[zhanglijian]

@Elasine
Hello
I have meet the same problems with you!!! How dou change the capability of GPU acccording computer, my GPU is Geforce 1080

[Elasine]

Sorry，I notice it too late. You could search the capability of GPU and compile it. On 9/21/2018 18:22，zhanglijian<[email protected]> wrote： @Elasine Hello I have meet the same problems with you!!! How dou change the capability of GPU acccording computer, my GPU is Geforce 1080 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.