Cannot find custom operator proposal Error whiling trying to export a Deformable-ConvNets which is based onmxnet into onnx

[hanzhaogang]

Description

I trained a model under Deformable-ConvNets(https://github.com/msracver/Deformable-ConvNets) which is based on mxnet. While trying to export it into onnx format, I hit this error: Cannot find custom operator proposal

it seems that mxnet cannot find teh proposal operator, which I confirmed that is a custom operator defined by the Deformable-ConvNets. This operator is defined as a subclass of mxnet.operator.CustomOp

I checked the Deformable-ConvNets rep and they claims that:

After MXNet@(commit ce2bca6) the offical MXNet support all operators for Deformable ConvNets.

So this operator should be supported.

Environment info (Required)

Package used (Python/R/Scala/Julia):
I'm using Python 3.6.6 & 3.7.1, both of them cannot avoid this issue.

Compiler (gcc/clang/mingw/visual studio):
visual studio 2017.

MXNet commit hash:
I copied the master branch.

Error Message:

D:>python onnx_conv_test.py
[14:15:48] C:\Jenkins\workspace\mxnet\mxnet\src\nnvm\legacy_json_util.cc:209: Lo
ading symbol saved by previous version v1.2.1. Attempting to upgrade...
Traceback (most recent call last):
File "onnx_conv_test.py", line 25, in
converted_model_path = onnx_mxnet.export_model(sym, params, [input_shape], n
p.float32, onnx_file)
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\site-p
ackages\mxnet\contrib\onnx\mx2onnx\export_model.py", line 75, in export_model
sym_obj, params_obj = load_module(sym, params)
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\site-p
ackages\mxnet\contrib\onnx\mx2onnx_export_helper.py", line 58, in load_module
sym, arg_params, aux_params = mx.model.load_checkpoint(model_name, num_epoch
s)
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\site-p
ackages\mxnet\model.py", line 437, in load_checkpoint
symbol = sym.load('%s-symbol.json' % prefix)
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\site-p
ackages\mxnet\symbol\symbol.py", line 2620, in load
check_call(_LIB.MXSymbolCreateFromFile(c_str(fname), ctypes.byref(handle)))
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\site-p
ackages\mxnet\base.py", line 252, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [14:15:48] C:\Jenkins\workspace\mxnet\mxnet\src\operator
custom\custom.cc:104: Check failed: CustomOperator::Get()->Find(params.op_type)
!= nullptr Cannot find custom operator proposal

Minimum reproducible example

just run below python scripts suggested by the mxnet offical guide: http://mxnet.incubator.apache.org/tutorials/onnx/export_mxnet_to_onnx.html

import mxnet as mx
import numpy as np
from mxnet.contrib import onnx as onnx_mxnet
sym = 'rfcn_voc-symbol.json'
params = 'rfcn_voc-0007.params'
input_shape = (1,3,224,224)
onnx_file = 'test.onnx'
converted_model_path = onnx_mxnet.export_model(sym, params, [input_shape], np.float32, onnx_file)

[vdantu]

@mxnet-label-bot add [ONNX, question, Operator]

[vandanavk]

@hanzhaogang Thank you for trying this out. The MXNet to ONNX exporter doesn't yet support operators such as Proposal, DeformablePSROIPooling, or Custom. ONNX doesn't have a specification for these operators either.

Having said that, the error that you are seeing is while loading the model, which is before export starts. Could you share the symbol.json and params files?

[hanzhaogang]

@vandanavk I am really sorry that due to the policy of our company I cann't share them here. I will try to reproduce this issue with some protrained models, after that I can provide those files here.

[Roshrini]

@hanzhaogang As Vandana mentioned before, we dont support exporting custom operators with ONNX so you wont be able to export the mentioned model.
But the error seems to be coming from "load_checkpoint" method which is unrelated to ONNX. Maybe there is something wrong with the model itself. Were you able to reproduce this issue with some other model as you mentioned before?

[assertdebug]

onnx.onnx_cpp2py_export.checker.ValidationError: Unrecognized attribute: spatial for operator BatchNormalization
==> Context: Bad node spec: input: "data" input: "bn_data_gamma" input: "bn_data_beta" input: "bn_data_moving_mean" input: "bn_data_moving_var" output: "bn_data" name: "bn_data" op_type: "BatchNormalization" attribute { name: "epsilon" f: 2e-05 type: FLOAT } attribute { name: "momentum" f: 0.9 type: FLOAT } attribute { name: "spatial" i: 0 type: INT }

[assertdebug]

onnx.onnx_cpp2py_export.checker.ValidationError: Unrecognized attribute: spatial for operator BatchNormalization
==> Context: Bad node spec: input: "data" input: "bn_data_gamma" input: "bn_data_beta" input: "bn_data_moving_mean" input: "bn_data_moving_var" output: "bn_data" name: "bn_data" op_type: "BatchNormalization" attribute { name: "epsilon" f: 2e-05 type: FLOAT } attribute { name: "momentum" f: 0.9 type: FLOAT } attribute { name: "spatial" i: 0 type: INT }

How to fixed it???

[wkcn]

Please set CXX_PROPOSAL to true in the yaml config of Deformable Conv.

However, I do not know whether onnx supports for proposal operator.

[x-dragonet]

@assertdebug I also have the problem, "ValidationError: Unrecognized attribute: spatial for operator BatchNormalization". How do you solve it?

[xudou3]

I also have the problem, "ValidationError: Unrecognized attribute: spatial for operator BatchNormalization". How do you solve it?

[souviksv]

How to fix this error???

ValidationError: Unrecognized attribute: spatial for operator BatchNormalization

==> Context: Bad node spec: input: "data" input: "bn_data_gamma" input: "bn_data_beta" input: "bn_data_moving_mean" input: "bn_data_moving_var" output: "bn_data" name: "bn_data" op_type: "BatchNormalization" attribute { name: "epsilon" f: 2e-05 type: FLOAT } attribute { name: "momentum" f: 0.9 type: FLOAT } attribute { name: "spatial" i: 0 type: INT }