Fused Op causes MXNetError

[leezu]

Description

After #15167 is merged, GluonNLP CI broke.

Error Message

[2019-11-06T06:44:48.223Z] mxnet.base.MXNetError: Error in operator Embedding_Dropout_Embedding_Dropout__FusedOp__contrib_arange_like__FusedOp_broadcast_lesser__FusedOp_broadcast_mul__FusedOp_broadcast_mul_expand_dims_broadcast_axis_Embedding__FusedOp_broadcast_add_Dropout_amp_cast_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected_Reshape_transpose_Reshape__contrib_div_sqrt_dim_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot__FusedOp_where_amp_cast_softmax__FusedOp_Dropout_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot_Reshape_transpose__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected_Reshape_transpose_Reshape__contrib_div_sqrt_dim_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot__FusedOp_where_amp_cast_softmax__FusedOp_Dropout_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot_Reshape_transpose__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected_Reshape_transpose_Reshape__contrib_div_sqrt_dim_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot__FusedOp_where_amp_cast_softmax__FusedOp_Dropout_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot_Reshape_transpose__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected_Reshape_transpose_Reshape__contrib_div_sqrt_dim_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot__FusedOp_where_amp_cast_softmax__FusedOp_Dropout_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot_Reshape_transpose__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected_Reshape_transpose_Reshape__contrib_div_sqrt_dim_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot__FusedOp_where_amp_cast_softmax__FusedOp_Dropout_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot_Reshape_transpose__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected_Reshape_transpose_Reshape__contrib_div_sqrt_dim_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot__FusedOp_where_amp_cast_softmax__FusedOp_Dropout_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot_Reshape_transpose__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected_Reshape_transpose_Reshape__contrib_div_sqrt_dim_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot__FusedOp_where_amp_cast_softmax__FusedOp_Dropout_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot_Reshape_transpose__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected_Reshape_transpose_Reshape__contrib_div_sqrt_dim_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot__FusedOp_where_amp_cast_softmax__FusedOp_Dropout_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot_Reshape_transpose__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected_Reshape_transpose_Reshape__contrib_div_sqrt_dim_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot__FusedOp_where_amp_cast_softmax__FusedOp_Dropout_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot_Reshape_transpose__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected_Reshape_transpose_Reshape__contrib_div_sqrt_dim_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot__FusedOp_where_amp_cast_softmax__FusedOp_Dropout_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot_Reshape_transpose__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected_Reshape_transpose_Reshape__contrib_div_sqrt_dim_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot__FusedOp_where_amp_cast_softmax__FusedOp_Dropout_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot_Reshape_transpose__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected_Reshape_transpose_Reshape__contrib_div_sqrt_dim_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot__FusedOp_where_zeros_like__FusedOpHelper_amp_cast_softmax__FusedOp_Dropout_amp_cast_amp_cast_FullyConnected_Reshape_transpose__FusedOp_batch_dot_Reshape_transpose__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_amp_cast_amp_cast_amp_cast_FullyConnected__FusedOp_amp_cast_amp_cast_FullyConnected_Dropout__FusedOp_amp_cast_amp_cast_LayerNorm_SequenceMask__FusedOp_amp_cast_amp_cast_FullyConnected_Activation_Dropout_amp_cast_amp_cast_amp_cast_FullyConnected__backward_FullyConnected__backward_amp_cast__backward_Dropout__backward_Activation__backward_FullyConnected__FusedOp__FusedOpHelper__backward_slice__backward_SequenceMask__backward_LayerNorm__FusedOp__backward_Dropout__backward_FullyConnected__FusedOp__backward_FullyConnected__FusedOp__backward_LayerNorm__FusedOp__backward_Dropout__backward_FullyConnected__FusedOpHelper__backward_amp_cast__FusedOpHelper__backward_reshape_transpose__backward_reshape_batch_dot_batch_dot__FusedOp__backward_Dropout__FusedOpHelper__backward_mul__FusedOpHelper__backward_amp_multicast__backward_amp_multicast: _Map_base::at

To Reproduce

git clone https://github.com/dmlc/gluon-nlp; cd gluon-nlp; pytest --color=yes -s scripts -k 'test_finetune_train[float16-WNLI-bert_12_768_12-2]'

Environment

https://pypi.org/project/mxnet-cu100/1.6.0b20191102/

[leezu]

@ptrendx

[sxjscience]

I suggest turn the fused_op off by default in the 1.6.0 release and announce it as experimental feature, or revert the PR. @szha @eric-haibin-lin @junrushao1994 @DickJC123 @wkcn @reminisce @haojin2 @TaoLv @marcoabreu What do you think?

[sxjscience]

@zhreshold

[wkcn]

I agree to turn the fused_op off by default until fused_op is stable.
The reason is that users couldn't use the 1.6.0 release if it is not compatible with their code.

[TaoLv]

+1

[ptrendx]

Isn't right now the period of finding those integration bugs and fixing them for 1.6 release? I will definitely look into this issue and fix it, not sure why you propose to turn the feature off by default?

[sxjscience]

@ptrendx I think we are already in a code-freeze status and the simplest fix is to turn it off by default. We could easily turn it on in 1.6.1 once we have confirmed that it has no impact in all the training scripts (there are plenty of them) and some may take time to run.

[ptrendx]

Ok, I sent a clarification email to dev@ as you are not actually the first person to reach out to me with this misunderstanding of code freeze. Code freeze is a period where bugs are found and fixed in order to polish the release and provide the best experience for the end users.

I treat the bugs about fusion with highest priority and will do my best to fix them. If I fail to address all issues before the time to make RC, then I agree it should be turned off by default and marked experimental.

[leezu]

I agree with @ptrendx, we should try to fix the bugs and ship the features if time allows.

[sxjscience]

I received the clarification email about the meaning of code freeze and I agree with @ptrendx that we should try to fix it these days and consider to turn it off by default if we fail to do so. BTW, what's the expected date for 1.6 RC?

[ptrendx]

I created a PR with a fix. @leezu, could you validate it?

[leezu]

@ptrendx thanks for the fix. Just confirmed it works.