Problems converting midasnet continued

[fricc33]

1. OS you are using: macOS

2. OS Architecture: Intel

3. Version of OpenVINO: openvino_2021.4.689

4. Version of TensorFlow: 2.6.0

5. Version of TensorRT: NA

6. Version of TFJS: NA

7. Version of coremltools NA

8. Version of ONNX: 1.10.1

9. Download URL for ONNX model: generated

10. Download URL for OpenVINO IR: generated

11. URL of the repository from which the transformed model was taken: https://github.com/isl-org/MiDaS

12. URL or source code for simple inference testing code: see attached zip file

13. Issue Details

I'd like to try this again as a follow up to #74, I apologize for my lack of clarity in my previous submission. My original message is at the bottom.

I tried processing your image in my environment and it worked just fine:

Maybe the issue is with the model weights file? I downloaded mine (MiDaS v2.1 large) from:

https://github.com/AlexeyAB/MiDaS/releases/download/midas_dpt/midas_v21-f6b98070.pt

Following the link on https://github.com/isl-org/MiDaS

Please let me know how we can work together on this.

Thank you very much for your patience and help,

• Fabio

openvino2tensorflow crashes with a tensor size mismatch to the input of layer Add 24.

I could trace the issue to the convolution and group convolution pad management logic (lines 626 and 1687)
Somehow the padding in the group convolution causes the output tensor to shrink by 2 pixels in both dimensions.

I don't have a fix, except that commenting out the code that introduces the padding allows the model to be generated, although the model will produce erroneous results.

I attach a zip file with the MiDaS distribution with all the ingredients to reproduce the issue:
Run the openvino2tensorflow_crash.sh script in the unzipped directory.
You will have to populate the weights directory with midas_v21-f6b98070.pt, and modify the script with the path to your OpenVINO distribution.

Midas-vino-crash.zip

[PINTO0309]

GroupConvolution bug fixes. a1a0c18

This tool has been upgraded to v1.22.2.
BTW, Docker works on Mac as well, so using a Docker container is highly recommended.

cd tf

wget -O ../weights/midas_v21-f6b98070.pt \
https://github.com/AlexeyAB/MiDaS/releases/download/midas_dpt/midas_v21-f6b98070.pt

python make_onnx_model.py

python3 -m onnxsim midas_v21-f6b98070.onnx midas_v21-f6b98070.onnx

docker pull pinto0309/openvino2tensorflow:latest

docker run -it --rm \
-v `pwd`:/home/user/workdir \
pinto0309/openvino2tensorflow:latest

cd workdir

$INTEL_OPENVINO_DIR/deployment_tools/model_optimizer/mo.py \
--input_model midas_v21-f6b98070.onnx \
--data_type FP32 \
--output_dir openvino/FP32

openvino2tensorflow \
--model_path openvino/FP32/midas_v21-f6b98070.xml \
--model_output_path openvino-tflite \
--output_no_quant_float32_tflite

tf_layers_dict: KerasTensor(type_spec=TensorSpec(shape=(1, 768, 1024, 1), dtype=tf.float32, name=None), name='tf.nn.relu_115/Relu:0', description="created by layer 'tf.nn.relu_115'")
====================================================================================
layer_type: Const
layer_id: 728
tf_layers_dict: (1,)
====================================================================================
layer_type: Squeeze
layer_id: 729
input_layer0: layer_id=727: KerasTensor(type_spec=TensorSpec(shape=(1, 768, 1024, 1), dtype=tf.float32, name=None), name='tf.nn.relu_115/Relu:0', description="created by layer 'tf.nn.relu_115'")
input_layer1: layer_id=728: Const(ndarray).shape (1,)
tf_layers_dict: KerasTensor(type_spec=TensorSpec(shape=(1, 768, 1024), dtype=tf.float32, name=None), name='tf.compat.v1.squeeze/Squeeze:0', description="created by layer 'tf.compat.v1.squeeze'")
====================================================================================
layer_type: Result
layer_id: 730
input_layer0: layer_id=729: KerasTensor(type_spec=TensorSpec(shape=(1, 768, 1024), dtype=tf.float32, name=None), name='tf.compat.v1.squeeze/Squeeze:0', description="created by layer 'tf.compat.v1.squeeze'")
tf_layers_dict: KerasTensor(type_spec=TensorSpec(shape=(1, 768, 1024), dtype=tf.float32, name=None), name='tf.identity/Identity:0', description="created by layer 'tf.identity'")
====================================================================================
TensorFlow/Keras model building process complete!
tflite Float32 convertion started ===================================================
tflite Float32 convertion complete! - openvino-tflite/model_float32.tflite
All the conversion process is finished! =============================================

MiDaS uses a lot of GroupConvolution in the depth direction, so I apply a workaround to decompose the converted tflite into Conv2D. This is because TensorFlow Lite does not support GroupConvolution. If you don't like the structure of the model, use onnx-tensorflow. However, onnx-tensorflow does not support clean conversion from NCHW to NHWC.

[fricc33]

Awesome! Thank you so much for your help and for your great tool.

I had actually figured out myself as well what the issue was with the padding not being consumed.

There is one more issue with MaxPool, which also allows for padding, see attached patch for a fix.
Now midasnet generated with openvino2tensorflow produces identical results to onnx-tensorflow, and the resulting model is twice as fast 😎

Thanks again, cheers,

• Fabio

MaxPool.patch.zip

[PINTO0309]

LGTM.

MaxPool bug fixes. c07667b

This tool has been upgraded to v1.22.3.

The problem of degraded accuracy of MiDaS, which had been a problem for a long time, may have been resolved. Thank you.
PINTO0309/PINTO_model_zoo#15

I would like to know what you can tell me in order to share your knowledge with other engineers. Can you tell me the inference time for the model generated by onnx-tensorflow and the approximate inference time for the model generated by openvino2tensorflow?

[PINTO0309]

Link:
tensorflow/tensorflow#40044

[fricc33]

Great, thanks!
BTW: I used to work in the TfLite GPU support team at Google Research, maybe I can give grouped convolutions a crack at a certain point. I run my model on the GPU so I would prioritize that.
Maybe we can close this issue for now :)

• Fabio