Enable unit tests for TVM ops for all cuda compute capabilities

[reminisce]

Description

As a result of #16777.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
• Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
• Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
• Code is well-documented:
• For user-facing API changes, API doc string has been updated.
• For new C++ functions in header files, their functionalities and arguments are documented.
• For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
• Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

• Feature1, tests, (and when applicable, API doc)
• Feature2, tests, (and when applicable, API doc)

Comments

• If this change is a backward incompatible change, why must this change be made.
• Interesting edge cases to note here

[ptrendx]

This affects 1.6, right? I encountered similar errors (CUDA_ERROR_INVALID_PTX) in testing of my unrelated PR #16796.

[reminisce]

@ptrendx This does not affect 1.6. We plan not to release TVM powered operators in 1.6. If you see the invalid ptx error, it's probably you invoked an operator powered by TVM, for example, mx.np.equal, mx.np.not_equal, mx.np.sum(a_boolean_ndarray), etc, on a sm_52 machine with float16 data inputs. Currently, you can skip those tests on such machines by checking mx.test_utils.is_op_runnable(). Even so, your tests will still be validated on a sm_52 machine with build flag USE_TVM_OP=0. After this PR is merged, we can treat builds w/ and w/o USE_TVM_OP equivalently.

[ptrendx]

Hmmm, so it looks like a slightly different error with similar result - I did not do anything with fp16 - this is my test:

@with_seed()
@use_np
def test_fusion_boolean_inputs_debug_tvm():
    from mxnet.gluon import HybridBlock

    class Foo(HybridBlock):
        def __init__(self, prefix=None, params=None):
            super(Foo, self).__init__(prefix=prefix, params=params)

        def hybrid_forward(self, F, valid_length):
            mask = (F.np.ones((10,)) < valid_length)
            return mask

    mx.npx.waitall()
    foo = Foo()
    foo.hybridize(static_alloc=True)
    out = foo(mx.np.ones((10,), ctx=mx.gpu()))
    mx.npx.waitall()

(note: I encountered it during fixing a bug in the pointwise fusion, that is why the test has fusion in the name, but it is actually a simplified version of the original test which does not involve fusion at all).
It invokes broadcast operator via TVM and it fails with

======================================================================

ERROR: test_fusion.test_fusion_boolean_inputs_debug_tvm

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest

    self.test(*self.arg)

  File "/work/mxnet/tests/python/gpu/../unittest/common.py", line 177, in test_new

    orig_test(*args, **kwargs)

  File "/work/mxnet/python/mxnet/util.py", line 321, in _with_np_shape

    return func(*args, **kwargs)

  File "/work/mxnet/python/mxnet/util.py", line 505, in _with_np_array

    return func(*args, **kwargs)

  File "/work/mxnet/tests/python/gpu/test_fusion.py", line 278, in test_fusion_boolean_inputs_debug_tvm

    mx.npx.waitall()

  File "/work/mxnet/python/mxnet/ndarray/ndarray.py", line 200, in waitall

    check_call(_LIB.MXNDArrayWaitAll())

  File "/work/mxnet/python/mxnet/base.py", line 278, in check_call

    raise MXNetError(py_str(_LIB.MXGetLastError()))

MXNetError: [00:40:04] /work/mxnet/3rdparty/tvm/src/runtime/module_util.cc:72: Check failed: ret == 0 (-1 vs. 0) : CUDAError: cuModuleLoadData(&(module_[device_id]), data_.c_str()) failed with error: CUDA_ERROR_INVALID_PTX

Stack trace:

  File "/work/mxnet/3rdparty/tvm/src/runtime/cuda/cuda_module.cc", line 111

  [bt] (0) /work/mxnet/python/mxnet/../../lib/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x32) [0x7f616bbd3d22]

  [bt] (1) /work/mxnet/python/mxnet/../../lib/libtvm_runtime.so(tvm::runtime::CUDAWrappedFunc::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*, void**) const+0x9c2) [0x7f61dd60b642]

  [bt] (2) /work/mxnet/python/mxnet/../../lib/libtvm_runtime.so(std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::PackedFunc tvm::runtime::detail::PackFuncVoidAddr_<8, tvm::runtime::CUDAWrappedFunc>(tvm::runtime::CUDAWrappedFunc, std::vector<tvm::runtime::detail::ArgConvertCode, std::allocator<tvm::runtime::detail::ArgConvertCode> > const&)::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#1}>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)+0xc4) [0x7f61dd60b954]

  [bt] (3) /work/mxnet/python/mxnet/../../lib/libtvm_runtime.so(TVMFuncCall+0x61) [0x7f61dd55fc61]



Stack trace:

  [bt] (0) /work/mxnet/python/mxnet/../../lib/libtvm_runtime.so(+0x52e1c) [0x7f61dd580e1c]

  [bt] (1) /work/mxnet/python/mxnet/../../lib/libmxnet.so(tvm::runtime::TVMOpModule::CallEx(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, tvm::runtime::TVMArgs) const+0xa6) [0x7f616e882e36]

  [bt] (2) /work/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::op::TVMBinaryBroadcastCompute::operator()(nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&)+0x40d) [0x7f616c7597ad]

  [bt] (3) /work/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::imperative::PushFCompute(std::function<void (nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&)> const&, nnvm::Op const*, nnvm::NodeAttrs const&, mxnet::Context const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::Resource, std::allocator<mxnet::Resource> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&)::{lambda(mxnet::RunContext)#1}::operator()(mxnet::RunContext) const+0x2b3) [0x7f616ef195c3]

  [bt] (4) /work/mxnet/python/mxnet/../../lib/libmxnet.so(std::_Function_handler<void (mxnet::RunContext), mxnet::imperative::PushFCompute(std::function<void (nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&)> const&, nnvm::Op const*, nnvm::NodeAttrs const&, mxnet::Context const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::Resource, std::allocator<mxnet::Resource> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&)::{lambda(mxnet::RunContext)#1}>::_M_invoke(std::_Any_data const&, mxnet::RunContext&&)+0x17) [0x7f616ef197f7]

  [bt] (5) /work/mxnet/python/mxnet/../../lib/libmxnet.so(std::_Function_handler<void (mxnet::RunContext, mxnet::engine::CallbackOnComplete), mxnet::engine::ThreadedEngine::BulkFlush()::{lambda(mxnet::RunContext, mxnet::engine::CallbackOnComplete)#1}>::_M_invoke(std::_Any_data const&, mxnet::RunContext&&, mxnet::engine::CallbackOnComplete&&)+0x1ec) [0x7f616f64747c]

  [bt] (6) /work/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::engine::ThreadedEngine::ExecuteOprBlock(mxnet::RunContext, mxnet::engine::OprBlock*)+0x995) [0x7f616f64b115]

  [bt] (7) /work/mxnet/python/mxnet/../../lib/libmxnet.so(void mxnet::engine::ThreadedEnginePerDevice::GPUWorker<(dmlc::ConcurrentQueueType)0>(mxnet::Context, bool, mxnet::engine::ThreadedEnginePerDevice::ThreadWorkerBlock<(dmlc::ConcurrentQueueType)0>*, std::shared_ptr<dmlc::ManualEvent> const&)+0x11d) [0x7f616f663a7d]

  [bt] (8) /work/mxnet/python/mxnet/../../lib/libmxnet.so(std::_Function_handler<void (std::shared_ptr<dmlc::ManualEvent>), mxnet::engine::ThreadedEnginePerDevice::PushToExecute(mxnet::engine::OprBlock*, bool)::{lambda()#4}::operator()() const::{lambda(std::shared_ptr<dmlc::ManualEvent>)#1}>::_M_invoke(std::_Any_data const&, std::shared_ptr<dmlc::ManualEvent>&&)+0x4e) [0x7f616f663d2e]

(link to the CI that failed: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-16796/6/pipeline)

[reminisce]

@ptrendx It fails because of the following line where the < operator targeted for sm_70 is invoked on a sm_52 machine. We currently only compile for sm_70 compute capability in the CI to avoid fp16 compilation problems in TVM.

F.np.ones((10,)) < valid_length

You can skip the test by decorating the test function as

@unittest.skipUnless(is_op_runnable(), 'skip cuda compute capability < 53 for TVM ops')
def test_fusion_boolean_inputs_debug_tvm():
    ...

[ptrendx]

Ok, if it is expected onmaster and does not affect 1.6 release then that's fine.