I'm sorry I've triggered an error in mxnet source code, how can I debug it? It seems like a check failure on custom operators, how can I find more details?

[Heermosi]

So far I'm working in linux dockers, the mxnet was built from source code
When I was running RoITransformer project with fpn training, the following error emerged.

File "experiments/fpn/fpn_end2end_train_test_RoITransformer.py", line 21, in < module>
train_end2end_rotbox_RoITransformer.main()
File "experiments/fpn/../../fpn/train_end2end_rotbox_RoITransformer.py", line 188, in main
config.TRAIN.begin_epoch, config.TRAIN.end_epoch, config.TRAIN.lr, config.TR AIN.lr_step)
File "experiments/fpn/../../fpn/train_end2end_rotbox_RoITransformer.py", line 181, in train_net
arg_params=arg_params, aux_params=aux_params, begin_epoch=begin_epoch, num_e poch=end_epoch)
File "experiments/fpn/../../fpn/core/module.py", line 989, in fit
self.update_metric(eval_metric, data_batch.label)
File "experiments/fpn/../../fpn/core/module.py", line 1081, in update_metric
self._curr_module.update_metric(eval_metric, labels)
File "experiments/fpn/../../fpn/core/module.py", line 672, in update_metric
self._exec_group.update_metric(eval_metric, labels)
File "experiments/fpn/../../fpn/core/DataParallelExecutorGroup.py", line 481, in update_metric
eval_metric.update(labels, texec.outputs)
File "/usr/local/lib/python2.7/site-packages/mxnet-1.6.0-py2.7.egg/mxnet/metri c.py", line 364, in update
metric.update(labels, preds)
File "experiments/fpn/../../fpn/core/metric.py", line 53, in update
pred_label = mx.ndarray.argmax_channel(pred).asnumpy().astype('int32')
File "/usr/local/lib/python2.7/site-packages/mxnet-1.6.0-py2.7.egg/mxnet/ndarr ay/ndarray.py", line 2506, in asnumpy
ctypes.c_size_t(data.size)))
File "/usr/local/lib/python2.7/site-packages/mxnet-1.6.0-py2.7.egg/mxnet/base. py", line 254, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [13:42:48] src/operator/custom/custom.cc:417: Check faile d: reinterpret_cast(params.info->callbacks[kCustomOpBackward])( ptrs.size(), const_cast<void**>(ptrs.data()), const_cast<int*>(tags.data()), rei nterpret_cast<const int*>(req.data()), static_cast(ctx.is_train), params.in fo->contexts[kCustomOpBackward]):
Stack trace:
[bt] (0) /usr/local/lib/python2.7/site-packages/mxnet-1.6.0-py2.7.egg/mxnet/li bmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x43) [0x7fd12dfa0133]
[bt] (1) /usr/local/lib/python2.7/site-packages/mxnet-1.6.0-py2.7.egg/mxnet/li bmxnet.so(+0x16d265f) [0x7fd12e65365f]
[bt] (2) /usr/local/lib/python2.7/site-packages/mxnet-1.6.0-py2.7.egg/mxnet/li bmxnet.so(+0x16db4f9) [0x7fd12e65c4f9]
[bt] (3) /usr/local/lib/python2.7/site-packages/mxnet-1.6.0-py2.7.egg/mxnet/li bmxnet.so(std::thread::_State_impl<std::thread::_Invoker<std::tuple<mxnet::op::c ustom::CustomOperator::SetNumThreads(int)::{lambda()#1}> > >::_M_run()+0xde) [0x 7fd12e66305e]
[bt] (4) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbd66f) [0x7fd174e7866f]
[bt] (5) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fd17b9786db]
[bt] (6) /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fd17aefc88f]

I've viewed the source code mentioned above, it looks like this:

CustomOperator::Get()->Push( [=]() { CHECK(reinterpret_cast<CustomOpFBFunc>(params.info->callbacks[kCustomOpBackward])( ptrs.size(), const_cast<void**>(ptrs.data()), const_cast<int*>(tags.data()), reinterpret_cast<const int*>(req.data()), static_cast<int>(ctx.is_train), params.info->contexts[kCustomOpBackward])); }, ctx, false, ctx.is_train, cpys, tags, output_tags, outputs, "_backward_" + params.op_type);
I cannot figure out what does this mean or which custom operator caused this.
Can any one give an advice on how to debug this???

[ddavydenko]

@mxnet-label-bot, add [Question, Debug]

[anirudh2290]

This may be related to #16365 . As a workaround. Can you please remove declare_backward_dependency for the custom op that is getting called and check : https://github.com/dingjiansw101/RoITransformer_DOTA/blob/master/fpn/operator_py/

[Heermosi]

This may be related to #16365 . As a workaround. Can you please remove declare_backward_dependency for the custom op that is getting called and check : https://github.com/dingjiansw101/RoITransformer_DOTA/blob/master/fpn/operator_py/

Hey, I'm on the way!
I'll post the result minutes after!
.....
I've got another error, I'm not sure it was corrected or not

mxnet.base.MXNetError: [14:54:20] src/storage/./pooled_storage_manager.h:161: cudaMalloc retry failed: out of memory

I've got report that the docker running the code cannot reveal the gpu usage properly, I'll redo the experiment later!
Thank you very much!

Can you tell how this happened?

[anirudh2290]

Can you tell how this happened?

It may be an issue with how storage type is assigned after invoking declare_backward_dependency callback : https://github.com/apache/incubator-mxnet/blob/master/src/operator/custom/custom.cc#L474 . I haven't had the time recently to dig deeper into this.

[Heermosi]

Can you tell how this happened?

It may be an issue with how storage type is assigned after invoking declare_backward_dependency callback : https://github.com/apache/incubator-mxnet/blob/master/src/operator/custom/custom.cc#L474 . I haven't had the time recently to dig deeper into this.

Ok, after I've got enough memory on this computer, the problem emerges again.
I'm now commented out all declare_backward_dependency with contents, and no use.
OK，I've deleted all such declarations, and no use
And, can you tell me how to debug it?
By the way, the exception was the same, it looks like this:

Traceback (most recent call last):
File "experiments/fpn/fpn_end2end_train_test_RoITransformer.py", line 21, in
train_end2end_rotbox_RoITransformer.main()
File "experiments/fpn/../../fpn/train_end2end_rotbox_RoITransformer.py", line 188, in main
config.TRAIN.begin_epoch, config.TRAIN.end_epoch, config.TRAIN.lr, config.TRAIN.lr_step)
File "experiments/fpn/../../fpn/train_end2end_rotbox_RoITransformer.py", line 181, in train_net
arg_params=arg_params, aux_params=aux_params, begin_epoch=begin_epoch, num_epoch=end_epoch)
File "experiments/fpn/../../fpn/core/module.py", line 989, in fit
self.update_metric(eval_metric, data_batch.label)
File "experiments/fpn/../../fpn/core/module.py", line 1081, in update_metric
self._curr_module.update_metric(eval_metric, labels)
File "experiments/fpn/../../fpn/core/module.py", line 672, in update_metric
self._exec_group.update_metric(eval_metric, labels)
File "experiments/fpn/../../fpn/core/DataParallelExecutorGroup.py", line 481, in update_metric
eval_metric.update(labels, texec.outputs)
File "/usr/local/lib/python2.7/site-packages/mxnet-1.6.0-py2.7.egg/mxnet/metric.py", line 364, in update
metric.update(labels, preds)
File "experiments/fpn/../../fpn/core/metric.py", line 53, in update
pred_label = mx.ndarray.argmax_channel(pred).asnumpy().astype('int32')
File "/usr/local/lib/python2.7/site-packages/mxnet-1.6.0-py2.7.egg/mxnet/ndarray/ndarray.py", line 2506, in asnumpy
ctypes.c_size_t(data.size)))
File "/usr/local/lib/python2.7/site-packages/mxnet-1.6.0-py2.7.egg/mxnet/base.py", line 254, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [18:28:21] src/operator/custom/custom.cc:417: Check failed: reinterpret_cast(params.info->callbacks[kCustomOpBackward])( ptrs.size(), const_cast<void**>(ptrs.data()), const_cast<int*>(tags.data()), reinterpret_cast<const int*>(req.data()), static_cast(ctx.is_train), params.info->contexts[kCustomOpBackward]):
Stack trace:
[bt] (0) /usr/local/lib/python2.7/site-packages/mxnet-1.6.0-py2.7.egg/mxnet/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x43) [0x7f019fb3d133]
[bt] (1) /usr/local/lib/python2.7/site-packages/mxnet-1.6.0-py2.7.egg/mxnet/libmxnet.so(+0x16d265f) [0x7f01a01f065f]
[bt] (2) /usr/local/lib/python2.7/site-packages/mxnet-1.6.0-py2.7.egg/mxnet/libmxnet.so(+0x16db4f9) [0x7f01a01f94f9]
[bt] (3) /usr/local/lib/python2.7/site-packages/mxnet-1.6.0-py2.7.egg/mxnet/libmxnet.so(std::thread::_State_impl<std::thread::_Invoker<std::tuplemxnet::op::custom::CustomOperator::SetNumThreads(int)::{lambda()#1} > >::_M_run()+0xde) [0x7f01a020005e]
[bt] (4) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbd66f) [0x7f022ca3166f]
[bt] (5) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f02335316db]
[bt] (6) /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f0232ab588f]

[TBag-ui]

Was a workaround ever found for this issue?

[anirudh2290]

@TBag-ui one workaround is mentioned here : #16365 (comment)