New Accuracy Layer on GPU interferes with training

[vlomonaco]

Issue summary

Using the "Accuracy" layer in the "Training net" on GPU breaks the training. The layer somehow interferes with the gradient. Loss explodes quickly and Train/Test Accuracies stall to 1.

Steps to reproduce

1. Download the the latest version of Caffe (commit 691febc)
2. Compile it with this Makefile.config
3. Run a network with Accuracy layer in the TRAIN phase.

My system configuration

Operating system: Ubuntu 14.04.5 LTS
CUDA version (if applicable): 7.0
CUDNN version (if applicable): 4.7
BLAS: libblas.so.3
Python version: 3.5

How to fix it

All these three solutions can fix it:

• Remove the Accuracy Layer from the "Training net", not problems with the phase: TEST.
• Change back-end to CPU.
• Rollback before commit 62e0c85 (which I suspect caused the issue).

[Noiredd]

I can reproduce this behavior for the GPU implementation. CPU does not seem to be affected. I'll be looking into this today; in the meantime, could you take a look too, @shaibagon?

EDIT: My suspicion after half an hour of tinkering with this: could it be that this memory actually is used for something? That is, we use it as a temporary memory but Caffe actually does propagate back from here?

[shaibagon]

@Noiredd Is it possible this causes the issue? I will look into it.

[Noiredd]

Removing the if and replacing with an unconditional NOT_IMPLEMENTED; did not change anything.

However, forcing:
caffe_gpu_set(bottom[0]->count(), Dtype(0), acc_data);
caffe_gpu_set(bottom[0]->count(), Dtype(0), counts);
at the end of Forward_gpu() fixes the problem, supporting my guess that Caffe indeed propagates from there.
This is not counter-intuitive. Think of intermediate classifiers - if we have a prediction blob A that is a bottom to one Accuracy layer but it's also a bottom to, let's say, InnerProduct, we want the IP to propagate. By reusing the blob's gradient memory, we effectively overrode the other gradients.

[shaibagon]

@Noiredd if you set counts and acc_data to zero - you are setting the gradients to zero. Thus, if caffe does propagate from there - you just killed the gradients.
I suppose it would require allocation of an internal blob to be used as a buffer.

[Noiredd]

@shaibagon Of course, this was just to prove that the problem is indeed there. I can come up with a fix in a while - unless you want to take it from here? Since you fathered this PR ;)

[shaibagon]

@Noiredd if it is okay with you, I'd appreciate if you can take it from here. I am not as available as I used to be for caffe :(

[Noiredd]

@vlomonaco Check PR #5987 - does it solve the issue for you?

[vlomonaco]

Hi @Noiredd thank you for the fix in less than 24hrs! It works!

[duygusar]

Hi, I have the exact same problem, somehow @Noiredd 's fix didn't work for me. Besides, I have my Accuracy layer only for the Test phase. I don't know why I am having this problem. My batchsize is not small and I have enough space, which rules out other reasons I have come across.