Can not infere a model with OpenVINO 2019R1

[bsekachev]

Hi,

In OpenVINO 2018 R5 I use MO to convert some model from TF OD API.

# Download and unpack the model
cd ${HOME} && \
wget -O model.tar.gz http://download.tensorflow.org/models/object_detection/faster_rcnn_inception_resnet_v2_atrous_coco_2018_01_28.tar.gz && \
tar -xzf model.tar.gz && rm model.tar.gz && \
mv faster_rcnn_inception_resnet_v2_atrous_coco_2018_01_28 ${HOME}/rcnn && cd ${HOME} && \
mv rcnn/frozen_inference_graph.pb rcnn/inference_graph.pb

# Convert it with MO
cd ${HOME}/rcnn/ && \
${INTEL_CVSDK_DIR}/deployment_tools/model_optimizer/mo.py --framework tf \
--data_type FP32 --input_shape [1,600,600,3] \
--input image_tensor --output detection_scores,detection_boxes,num_detections \
--tensorflow_use_custom_operations_config ${INTEL_CVSDK_DIR}/deployment_tools/model_optimizer/extensions/front/tf/faster_rcnn_support.json \
--tensorflow_object_detection_api_pipeline_config pipeline.config --input_model inference_graph.pb

Next I infer the converted model with 2018R5 IE:

# image_np is [3, 600, 600]
prediction = executable_network.infer(inputs={input_blob_name: image_np[np.newaxis, ...]})[output_blob_name][0][0]

To sum up pipeline works well in 2018R5 IE. But something wrong in 2019R1.
Conversion is ok, but inference failed with an error:

prediction = executable_network.infer(inputs={input_blob_name: image_np[np.newaxis, ...]})[output_blob_name][0][0]
File "ie_api.pyx", line 146, in openvino.inference_engine.ie_api.ExecutableNetwork.infer
File "ie_api.pyx", line 179, in openvino.inference_engine.ie_api.InferRequest.infer
File "ie_api.pyx", line 181, in openvino.inference_engine.ie_api.InferRequest.infer
File "ie_api.pyx", line 233, in openvino.inference_engine.ie_api.InferRequest._fill_inputs
ValueError: could not broadcast input array from shape (3,600,600) into shape (1,3)

In addition, model converted with OpenVINO 2018R5 MO works well with OpenVINO 2019R1 IE.

[shubha-ramani]

Dear @bsekachev,
In OpenVino 2019 R1 there is also a faster_rcnn_support_api_v1.7.json and a faster_rcnn_support_api_v1.10.json. The one you use depends on the tensorflow API version used in the model.

Did you follow the version rules in the mo tf object detection api doc ?

It seems like you did follow the rules because otherwise probably MO would have failed to generate IR, but I would just like to make sure.

Thanks,

Shubha

[bsekachev]

@shubha-ramani
I use faster rcnn and as far as I understood, there aren't significant differences between these files and both can be used for any API version. Right?

faster_rcnn_support_api_v1.7.json — for Faster R-CNN topologies trained manually using the TensorFlow* Object Detection API version 1.7.0 or higher
faster_rcnn_support_api_v1.10.json — for Faster R-CNN topologies trained manually using the TensorFlow* Object Detection API version 1.7.0 or higher

It seems to me, my frozen graph have been got with TF API v 1.12.0 as described here

UPD:
I read it carefully and it seems, for frozen graphs here is only faster_rcnn_support.json .
faster_rcnn_support_api_* aren't used for it.

[shubha-ramani]

Dear @bsekachev,

if you use a diff tool to compare the different versions of faster_rcnn*.json within the C:\Program Files (x86)\IntelSWTools\openvino_2019.1.087\deployment_tools\model_optimizer\extensions\front directory you will see that they are different. If they weren't different there is no need to have separate versions.

If you are using TF API v 1.12.0, the aforementioned online MO doc says this:

faster_rcnn_support_api_v1.10.json — for Faster R-CNN topologies trained manually using the TensorFlow* Object Detection API version 1.7.0 or higher

Which means you definitely should be using faster_rcnn_support_api_v1.10.json.

Hope it helps,

Thanks,

Shubha

[shubha-ramani]

Dear @bsekachev ,

Additionally, if a frozen graph is not readily available for download you can always make one yourself using Tensorflow code and tools (it's fairly easy to do).

Thanks,

Shubha

[bsekachev]

@shubha-ramani
I have downloaded frozen graph and the documentation tells me to use exactly the file faster_rcnn_support.json and not other.

What about faster_rcnn_support_api_* files, of course I have guessed they are different, but I told about there aren't any differences in them descriptions in the documentation.

[shubha-ramani]

@bsekachev Seems like you found a bug though. I will reproduce and file a bug - the IE should not throw an exception in this case. Please tell me which OpenVino sample you are using for inference, or you can also attach the script you use here.

Thanks,

Shubha

[bsekachev]

@shubha-ramani

Error occurs in this line.

Thanks.

[shubha-ramani]

Dear @bsekachev, I am still investigating this. I tried and faster_rcnn_support_api_v1.7.json and faster_rcnn_support_api_v1.10.json cause MO to fail anyway. The only one which generates IR is the one you used - just faster_rcnn_support.json . If you look at the date of the model, it's pretty old (about 1.5 years) so the fact that an older API was used is quite believable. 

[Drizshko]

Looks like model has 2 input, one for image with shape [1,3,600,600] and second one for image info with shape [1,3].

Instead of doing this: input_blob_name = next(iter(network.inputs)) you need to iterate thru network inputs and find one with 4 dimensions - this will be an image tensor, use it as input_blob_name. Second one will be image info tensor, fill it like this:

info[0]=image_width;
info[1]=image_height;
info[2]=1;

[shubha-ramani]

Dear @bsekachev,
Unfortunately your code is pretty complicated and since I support all the forums I don't have the bandwidth to debug or parse it. Please use one of our OpenVino samples to make a compact sample which demonstrates your issue. Basically, I would like a "main.cpp" which I can drop into our samples infrastructure - one that demonstrates your issue.

I did successfully generate IR using your command above for mask_rcnn_inception_resnet_v2_atrous_coco_2018_01_28

Thanks,

Shubha

[shubha-ramani]

Dear @bsekachev,
Please accept my apologies. I just remembered that we have our own mask_rcnn_demo.exe which I can run ! And indeed, there is no problem. Here is the output:
`
C:\Users\sdramani\Documents\Intel\OpenVINO\inference_engine_samples_build_2017\intel64\Release>mask_rcnn_demo.exe -m c:\users\sdramani\Downloads\mask_rcnn_inception_resnet_v2_atrous_coco_2018_01_28\frozen_inference_graph.xml -i c:\users\sdramani\Downloads\pics\horse.bmp
InferenceEngine:
API version ............ 1.6
Build .................. 22443
[ INFO ] Parsing input parameters
[ INFO ] Files were added: 1
[ INFO ] c:\users\sdramani\Downloads\pics\horse.bmp
[ INFO ] Loading plugin

    API version ............ 1.6
    Build .................. 22443
    Description ....... MKLDNNPlugin

[ INFO ] Loading network files
[ INFO ] Preparing input blobs
[ INFO ] Network batch size is 1
[ INFO ] Prepare image c:\users\sdramani\Downloads\pics\horse.bmp
[ WARNING ] Image is resized from (500, 333) to (800, 800)
[ INFO ] Preparing output blobs
[ INFO ] Loading model to the plugin
[ INFO ] Setting input data to the blobs
[ INFO ] Start inference (1 iterations)

Average running time of one iteration: 18987.7 ms

[ INFO ] Processing output blobs
[ INFO ] Detected class 19 with probability 0.999372 from batch 0: [201.489, 114.285], [403.88, 309.797]
[ INFO ] Image out0.png created!
[ INFO ] Execution successful

The output of this sample is the image with segmentation applied.

@bsekachev,

Please compare your code to the mask_rcnn_demo code.

Thanks,

Shubha

`

[alalek]

MO 2018R5 generates model IRs with these inputs/outputs (see the first post about used parameters of MO):

network.inputs = ['image_tensor']
network.outputs = ['detection_output']

MO 2019R1 IRs:

network.inputs = ['image_tensor', 'image_info']
network.outputs = ['detection_output']

Tested using Python code from IE 2019R1:

from openvino.inference_engine import IENetwork
network = IENetwork.from_ir(model = model, weights = weights)
print('network.inputs = ' + str(list(network.inputs)))
print('network.outputs = ' + str(list(network.outputs)))

So, incompatible change is going from MO.
It starts adding of extra input for TF OD models since 2019R1.
Related MO lines.

Not sure if this can be changed via configuration files. Need to update inference code of application (this is not good from compatibility point of view).

---

At least, we should fix error message: add name of the problem layer.
Lets start error reporting from missed layers. Passed 1 layer data, but expected 2 => fail fast with appropriate message.
Actual message is could not broadcast input array from shape (3,600,600) into shape (1,3) (layer name is missing in this message).

[bsekachev]

@alalek
Okay, I understood. Thanks.