test_executor.test_bind has fixed seed that can mask flakiness

[szha]

The unit test in title have been using fixed seed to mask flakiness. Suggested action:

1. Evaluate whether the test is flaky without fixed seed. If not, remove seed. Else move to 2
2. If test is flaky, determine whether it's an actual uncaught edge case. If so, fix the operator. Else move to 3
3. If numerical instability is inevitable, adjust tolerance level appropriately.

[apeforest]

Thanks for filing this issue. We will investigate this Flaky test

[kalyc]

Able to reproduce flaky test error -

def test_bind():
        def check_bind(disable_bulk_exec):
            if disable_bulk_exec:
                prev_bulk_inf_val = mx.test_utils.set_env_var("MXNET_EXEC_BULK_EXEC_INFERENCE", "0", "1")
                prev_bulk_train_val = mx.test_utils.set_env_var("MXNET_EXEC_BULK_EXEC_TRAIN", "0", "1")
    
            nrepeat = 10
            maxdim = 4
            for repeat in range(nrepeat):
                for dim in range(1, maxdim):
                    check_bind_with_uniform(lambda x, y: x + y,
                                            lambda g, x, y: (g, g),
                                            dim)
                    check_bind_with_uniform(lambda x, y: x - y,
                                            lambda g, x, y: (g, -g),
                                            dim)
                    check_bind_with_uniform(lambda x, y: x * y,
                                            lambda g, x, y: (y * g, x * g),
                                            dim)
                    check_bind_with_uniform(lambda x, y: x / y,
                                            lambda g, x, y: (g / y, -x * g/ (y**2)),
                                            dim)
    
                    check_bind_with_uniform(lambda x, y: np.maximum(x, y),
                                            lambda g, x, y: (g * (x>y), g * (y>x)),
                                            dim,
                                            sf=mx.symbol.maximum)
                    check_bind_with_uniform(lambda x, y: np.minimum(x, y),
                                            lambda g, x, y: (g * (x<y), g * (y<x)),
                                            dim,
                                            sf=mx.symbol.minimum)
            if disable_bulk_exec:
               mx.test_utils.set_env_var("MXNET_EXEC_BULK_EXEC_INFERENCE", prev_bulk_inf_val)
               mx.test_utils.set_env_var("MXNET_EXEC_BULK_EXEC_TRAIN", prev_bulk_train_val)
    
>       check_bind(True)

test_executor.py:117: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
test_executor.py:108: in check_bind
    sf=mx.symbol.maximum)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

uf = <function <lambda> at 0x1078889b0>, gf = <function <lambda> at 0x10e2852a8>, dim = 3, sf = <function maximum at 0x10c7c3b90>, lshape = (3, 1, 1), rshape = (3, 1, 1)

    def check_bind_with_uniform(uf, gf, dim, sf=None, lshape=None, rshape=None):
        """check function consistency with uniform random numbers"""
        shape = tuple(np.random.randint(1, int(1000**(1.0/dim)), size=dim))
        lhs = mx.symbol.Variable('lhs')
        rhs = mx.symbol.Variable('rhs')
        if sf is not None:
            ret = sf(lhs, rhs)
        else:
            ret = uf(lhs, rhs)
    
        assert ret.list_arguments() == ['lhs', 'rhs']
        lshape = shape if lshape is None else lshape
        rshape = shape if rshape is None else rshape
    
        lhs_arr = mx.nd.array(np.random.uniform(-1, 1, lshape))
        rhs_arr = mx.nd.array(np.random.uniform(-1, 1, rshape))
        lhs_grad = mx.nd.empty(lshape)
        rhs_grad = mx.nd.empty(rshape)
        executor = ret.bind(mx.Context('cpu'),
                            args=[lhs_arr, rhs_arr],
                            args_grad=[lhs_grad, rhs_grad])
    
        exec3 = ret.bind(mx.Context('cpu'),
                         args=[lhs_arr, rhs_arr])
    
    
        exec4 = ret.bind(mx.Context('cpu'),
                         args={'rhs': rhs_arr, 'lhs': lhs_arr},
                         args_grad={'lhs': lhs_grad, 'rhs': rhs_grad})
    
        executor.forward()
        exec3.forward()
        exec4.forward()
        out2 = executor.outputs[0].asnumpy()
        out1 = uf(lhs_arr.asnumpy(), rhs_arr.asnumpy())
        out3 = exec3.outputs[0].asnumpy()
        out4 = exec4.outputs[0].asnumpy()
        assert reldiff(out1, out2) < 1e-6
        assert reldiff(out1, out3) < 1e-6
        assert reldiff(out1, out4) < 1e-6
        # test gradient
        out_grad = mx.nd.array(np.ones(out2.shape))
        lhs_grad2, rhs_grad2 = gf(out_grad.asnumpy(),
                                  lhs_arr.asnumpy(),
                                  rhs_arr.asnumpy())
        executor.backward([out_grad])
    
>       assert reldiff(lhs_grad.asnumpy(), lhs_grad2) < 1e-6
E       assert nan < 1e-06
E        +  where nan = reldiff(array([[[ 0.]],\n\n       [[ 0.]],\n\n       [[ 0.]]], dtype=float32), array([[[ 0.]],\n\n       [[ 0.]],\n\n       [[ 0.]]], dtype=float32))
E        +    where array([[[ 0.]],\n\n       [[ 0.]],\n\n       [[ 0.]]], dtype=float32) = <bound method NDArray.asnumpy of \n[[[ 0.]]\n\n [[ 0.]]\n\n [[ 0.]]]\n<NDArray 3x1x1 @cpu(0)>>()
E        +      where <bound method NDArray.asnumpy of \n[[[ 0.]]\n\n [[ 0.]]\n\n [[ 0.]]]\n<NDArray 3x1x1 @cpu(0)>> = \n[[[ 0.]]\n\n [[ 0.]]\n\n [[ 0.]]]\n<NDArray 3x1x1 @cpu(0)>.asnumpy

test_executor.py:77: AssertionError

[kalyc]

For this test when seed is not set to 0, reldiff is NaN as diff & norm = 0.0 & hence we get an assertion error.
When seed is set to 0, we dont get this error as the above mentioned values are non zero.

[roywei]

after change with #11978 , this test still fails with:

MXNET_TEST_SEED=1875642038 nosetests --logging-level=DEBUG --verbose -s test_executor.py:test_bind

error message:

======================================================================
FAIL: test_executor.test_bind
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/ubuntu/roy/incubator-mxnet/tests/python/unittest/common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/roy/incubator-mxnet/tests/python/unittest/test_executor.py", line 111, in test_bind
    check_bind(True)
  File "/home/ubuntu/roy/incubator-mxnet/tests/python/unittest/test_executor.py", line 102, in check_bind
    sf=mx.symbol.maximum)
  File "/home/ubuntu/roy/incubator-mxnet/tests/python/unittest/test_executor.py", line 71, in check_bind_with_uniform
    assert_almost_equal(lhs_grad.asnumpy(), lhs_grad2, rtol=1e-5, atol=1e-5)
  File "/usr/local/lib/python3.5/dist-packages/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 100000.000000 exceeds tolerance rtol=0.000010, atol=0.000010.  Location of maximum error:(5, 17), a=1.000000, b=0.000000
 a: array([[0., 0., 1., ..., 0., 0., 1.],
       [0., 1., 1., ..., 0., 0., 0.],
       [1., 1., 0., ..., 1., 0., 1.],...
 b: array([[0., 0., 1., ..., 0., 0., 1.],
       [0., 1., 1., ..., 0., 0., 0.],
       [1., 1., 0., ..., 1., 0., 1.],...
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1585133920 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1875642038 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 0.033s

FAILED (failures=1)

[kalyc]

On further examination, @roywei's changes are failing on this check -
check_bind_with_uniform(lambda x, y: np.maximum(x, y), lambda g, x, y: (g * (x>y), g * (y>x)), dim, sf=mx.symbol.maximum)
on lines - https://github.com/apache/incubator-mxnet/blob/master/tests/python/unittest/test_executor.py#L99-L102

[roywei]

@kalyc my change in PR #11978 is independent from the failing. assert_almost_equal is a better way to compare values. It should be failing here, because the unit test is wrong.

Root cause here is this test does not cover the corner case when x=y in maximum and minimum.
See detailed explanation in #12016

This test should be fixed by #12016