Solve memory inefficiency in RNNs

[atmguille]

As discussed with @mattdangerw, I am opening this PR to fix #16113 memory inefficiency in RNNs.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

For more information, open the CLA check for this pull request.

[fchollet]

Thanks for the PR!

[fchollet]

LGTM, thanks

[fchollet]

There are test failures, please take a look: https://source.cloud.google.com/results/invocations/919aaccc-7dd1-4d0e-9912-0596f18e60c1/targets/keras%2Fgithub%2Fubuntu%2Fgpu%2Fpresubmit/log

[atmguille]

@fchollet just saw the test log. I think it comes from the following:
LSTM and GRU have different functions to execute in CPU and GPU (the other RNNs layers don't). CPU version calls backend.rnn (the function I updated), but GPU calls tf.raw_ops.CudnnRNN. I updated the CPU version to properly call the new version of backend.rnn, but the GPU one will always return outputs as tf.raw_ops.CudnnRNN always returns the full array.

The error happens in the tests that combine CPU and GPU environments.

I see three options to solve this:

1. Deactivate new functionality in LSTM and GRU. As mentioned in the original issue Memory inefficiency in RNNs #16113, the layers most affected by the current behavior are convolutional RNNs, for which the update works perfectly.
2. Although tf.raw_ops.CudnnRNN returns outputs, the GPU method can still return outputs=[last_output]. The memory advantage will not be available in GPU for LSTM and GRU, but the return format from both CPU and GPU would be consistent.
3. Update tf.raw_ops.CudnnRNN in a similar way than backend.rnn. However, I have not been able to find the source code of that function in the tensorflow repository.

I think the ideal solution would be 3. For the time being, I would go for 2. Let me know what you think!

PS: I had executed the tests locally with no errors. I have re-executed them again and no errors were found, as can be seen in this notebook.

[fchollet]

Let's do option 2. Option 3. is not realistically doable given that the computation is delegated to cuDNN.

[atmguille]

I believe it should be fixed now. I have executed all tests locally and there are no errors. However, there were no errors also before this fix, as I mentioned in my last comment. Let's see

Thanks again for your time @fchollet

[fchollet]

LGTM, thanks!

[atmguille]

@fchollet I saw how the tests you requested worked but the tests that have just executed do not.
The log shows the following:

Maybe floating point operations errors being not deterministic? That would explain why your tests passed and these do not. Should we try to run them again? Let me know how I can help

Edit: I have checked the code of the test that fails and apparently it has nothing to do with my updates. The model being tested is this one

[fchollet]

I think this is possibly a case of flaky test. Let's try rerunning the tests.

[atmguille]

The tests have passed! I am not sure if you need to approve my last review request, as the PR is "Approved by Reviewer" but it also shows that there is 1 pending reviewer.

Anyway, thanks a lot for your time! It was a good learning experience as my first contribution @fchollet