Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

test_loss.test_squared_hinge_loss has fixed seed that can mask flakiness #11702

Closed
szha opened this issue Jul 13, 2018 · 3 comments
Closed

test_loss.test_squared_hinge_loss has fixed seed that can mask flakiness #11702

szha opened this issue Jul 13, 2018 · 3 comments

Comments

@szha
Copy link
Member

szha commented Jul 13, 2018

The unit test in title have been using fixed seed to mask flakiness. Suggested action:

  1. Evaluate whether the test is flaky without fixed seed. If not, remove seed. Else move to 2
  2. If test is flaky, determine whether it's an actual uncaught edge case. If so, fix the operator. Else move to 3
  3. If numerical instability is inevitable, adjust tolerance level appropriately.
@apeforest
Copy link
Contributor

Thanks for filing this issue. We will investigate this Flaky test

@zachgk
Copy link
Contributor

zachgk commented Aug 3, 2018

Ran tests/python/unittest/test_loss.py:test_squared_hinge_loss and tests/python/gpu/test_operator_gpu.py:test_squared_hinge_loss 10k times each and could not reproduce flakiness
Raised a PR #12017 to remove the fixed seed

CPU Run:

ubuntu@ip-172-31-27-232:~/incubator-mxnet$ MXNET_TEST_COUNT=10000 nosetests --logging-level=DEBUG --verbose -s tests/python/unittest/test_loss.py:test_squared_hinge_loss

[DEBUG] 9999 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=711236678 to reproduce.
[DEBUG] 10000 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1402836224 to reproduce.
ok

----------------------------------------------------------------------
Ran 1 test in 7828.158s

OK

GPU Run:

ubuntu@ip-172-31-27-232:~/incubator-mxnet$ MXNET_TEST_COUNT=10000 nosetests --logging-level=DEBUG --verbose -s tests/python/gpu/test_operator_gpu.py:test_squared_hinge_loss

[DEBUG] 9999 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=330147794 to reproduce.
[DEBUG] 10000 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=608869560 to reproduce.
ok

----------------------------------------------------------------------
Ran 1 test in 8022.626s

OK

@haojin2
Copy link
Contributor

haojin2 commented Aug 16, 2018

@sandeep-krishnamurthy Related PR is merged, should be good to close.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

5 participants