Flaky test: test_operator_gpu.test_rnnrelu_bidirectional

[larroy]

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-11382/7/pipeline/

======================================================================

FAIL: test_operator_gpu.test_rnnrelu_bidirectional

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest

    self.test(*self.arg)

  File "/usr/local/lib/python3.5/dist-packages/nose/util.py", line 620, in newfunc

    return func(*arg, **kw)

  File "/work/mxnet/tests/python/gpu/../unittest/common.py", line 157, in test_new

    orig_test(*args, **kwargs)

  File "/work/mxnet/tests/python/gpu/../unittest/test_operator.py", line 207, in test_rnnrelu_bidirectional

    check_rnn_consistency(fused, stack, T, N, I, H, 'add')

  File "/work/mxnet/tests/python/gpu/../unittest/test_operator.py", line 65, in check_rnn_consistency

    assert_allclose(mod1.get_input_grads()[0].asnumpy(), mod2.get_input_grads()[0].asnumpy(), rtol=1e-2, atol=1e-4)

  File "/usr/local/lib/python3.5/dist-packages/numpy/testing/nose_tools/utils.py", line 1396, in assert_allclose

    verbose=verbose, header=header, equal_nan=equal_nan)

  File "/usr/local/lib/python3.5/dist-packages/numpy/testing/nose_tools/utils.py", line 779, in assert_array_compare

    raise AssertionError(msg)

AssertionError: 

Not equal to tolerance rtol=0.01, atol=0.0001



(mismatch 0.01999999999999602%)

 x: array([[[ 7.377952e-03,  1.249200e-02, -2.486279e-03, ...,

         -1.402252e-03, -5.022895e-04, -7.681327e-04],

        [ 7.553479e-03,  8.460643e-03,  2.067674e-03, ...,...

 y: array([[[ 7.377951e-03,  1.249200e-02, -2.486280e-03, ...,

         -1.402253e-03, -5.022900e-04, -7.681326e-04],

        [ 7.553477e-03,  8.460643e-03,  2.067674e-03, ...,...

-------------------- >> begin captured logging << --------------------

common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1834194485 to reproduce.

--------------------- >> end captured logging << ---------------------

[frankfliu]

@larroy Thank you for reporting the Flaky test, this is added to the to do test improvements

[pengzhao-intel]

Looks this is a precision issue and the diff from GPU/CPU are very tiny. @lihaofd please take a look what's the max diff in the results array?

[haojin2]

@sandeep-krishnamurthy Related PR is merged, should be good to close.