-
Notifications
You must be signed in to change notification settings - Fork 6.8k
NightlyTestsForBinaries tutorials test broken #15374
Comments
Hey, this is the MXNet Label Bot. |
The test was broken by this PR #15170: [MXNET-1413] Adding Large Tensor support for sort operators. #if USE_INT64_TENSOR_SIZE == 1
.set_default(mshadow::kInt64)
#else
.set_default(mshadow::kInt32)
#endif This has caused a problem in gluon-cv in rank = (cls_loss * (pos - 1)).argsort(axis=1).argsort(axis=1)
hard_negative = rank < nd.maximum(self._min_hard_negatives, pos.sum(axis=1)
* self._negative_mining_ratio).expand_dims(-1)
|
@mxnet-label-bot add labels Test |
@access2rohit could you take a look and say whether we should revert your PR or recommend GluonCV to adopt to new behaviour? |
In my opinion it's not very good that |
@mxnet-label-bot add [Test] |
#15360 will hopefully fix this. |
@lebeg I will merge my PR once unix tests pass. Currently they are failing on unix CI/CD build, which is independent of my code change. |
I've restarted both unix verification jobs. |
Unfortunately this is still broken as of Jul 8th. |
Still failing as of Jul 15th: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/NightlyTestsForBinaries/detail/master/379/pipeline |
Still failing as of the 19th http://jenkins.mxnet-ci.amazon-ml.com/job/NightlyTestsForBinaries/job/master/ |
@access2rohit Can you help to check if the nightly test failure is still related to your topk change? |
Ran amp tutorial manually and found no error Output: |
`$ python -c "from mxnet.runtime import Features; print(Features())" [✔ CUDA, ✔ CUDNN, ✖ NCCL, ✖ CUDA_RTC, ✖ TENSORRT, ✔ CPU_SSE, ✔ CPU_SSE2, ✔ CPU_SSE3, ✔ CPU_SSE4_1, ✔ CPU_SSE4_2, ✖ CPU_SSE4A, ✔ CPU_AVX, ✖ CPU_AVX2, ✖ OPENMP, ✖ SSE, ✔ F16C, ✖ JEMALLOC, ✔ BLAS_OPEN, ✖ BLAS_ATLAS, ✖ BLAS_MKL, ✖ BLAS_APPLE, ✖ LAPACK, ✖ MKLDNN, ✖ OPENCV, ✖ CAFFE, ✖ PROFILER, ✖ DIST_KVSTORE, ✖ CXX14, ✔ INT64_TENSOR_SIZE, ✖ SIGNAL_HANDLER, ✖ DEBUG, ✖ TVM_OP]` |
The nightly test for tutorials is broken, in particular test_tutorials.test_amp
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/NightlyTestsForBinaries/detail/master/355/pipeline
The text was updated successfully, but these errors were encountered: