Is MatMul still not supported?

[koodzi]

Model Optimizer version: 2019.1.0-341-gc9b66a2

I have got a model that uses tf.matmul and in the previous version mo_tf.py returned an error MatMul cannot be converted to IE IR but in the new version, it converts to IR successfully [ SUCCESS ] Generated IR model.

However during loading time in IENetwork I get:
RuntimeError: Error reading network: in Layer X: trying to connect an edge to nonexisting output port: 98.2

Indeed In XML file, there is no layer 98 at all: the matmul operation is missing. It appears converter removed unsupported operation without warning. After a search, I found supported operations page but I do not understand how can I replace matmul with Dense (FullConnected) layer.

How can I enable tf.matmul in my graph?
Can I just edit XML? Are there examples with explanation? Maybe IR docs?

<layer id="98" name="classifier_scope/dense_1/MatMul" precision="FP16" type="FullyConnected">
			<data out-size="28"/>
			<input>
				<port id="0">
					<dim>1</dim>
					<dim>64</dim>
				</port>
			</input>
			<output>
				<port id="3">
					<dim>1</dim>
					<dim>28</dim>
				</port>
			</output>
			<blobs>
				<weights offset="186995100" size="3584"/>
				<biases offset="186998684" size="56"/>
			</blobs>
		</layer>

[shubha-ramani]

Dear @konradNowicki
MatMul is fully supported by the Model Optimizer. On the Inference Engine end the MatMul is expressed as FullyConnected or Gemm layer. The FullyConnected is a special case of the Gemm when one of the inputs is constant. Can you kindly send me (or attach) your particular model ? What code are you using for inference ?

Thanks,

Shubha

[koodzi]

Hi, I manage to boil down the problem to just a few lines and I have added some code for easy running and debugging.

import os

import tensorflow as tf
from tensorflow.layers import Dense
from tensorflow.python.framework import graph_io
import numpy as np

import subprocess
slim = tf.contrib.slim

def graph_and_freeze():

    def the_graph(in1, in2): # in1=shape(?, 5, 64), in2=shape(?, 5, 1)
        """Graph definition (simplified)"""

        net = Dense(5, activation=None, use_bias=False)(in1)  # = shape(?, 5, 5)
        net = net + in2 # = shape(?, 5, 5)
        net = Dense(1, activation=None, use_bias=False)(net)  # = shape(?, 5, 1)
        net = tf.squeeze(net, axis=2)  # = shape(?, 5)
        net = tf.nn.softmax(net, axis=-1)  # = shape(?, 5)
        out1 = tf.reshape(net, [-1, 1, net.shape[-1]], name="out1") # = shape(?, 1, 5)

        out2 = tf.matmul(out1, in1)  # = shape(?, 1, 64) (THIS OPERATION IS MISSING IN XML)
        out2 = tf.reshape(out2, [-1, out2.shape[-1]], name="out2")  # = shape(?, 64)

        return out1, out2

    in1 = tf.placeholder(tf.float32, (None, 5, 64), "in1")
    in2 = tf.placeholder(tf.float32, (None, 5, 1), "in2")
    out1, out2 = the_graph(in1, in2)

    with tf.Session() as sess:
        print(sess.graph_def)

        tf.global_variables_initializer().run()
        batch_size = 10

        sess.run([out1, out2], feed_dict={
            in1: np.random.rand(batch_size, *in1.get_shape().as_list()[1:]),
            in2: np.random.rand(batch_size, *in2.get_shape().as_list()[1:])
        })

        frozen = tf.graph_util.convert_variables_to_constants(sess, sess.graph_def, ['out1', 'out2'])
        graph_io.write_graph(frozen, './', 'test.frozen.pb', as_text=False)


def translate():
    open_vino = os.environ['INTEL_OPENVINO_DIR']
    if open_vino is None:
        print("\n\nInitialize INTEL_OPENVINO environement (setvars.sh) `\n\n\n")
        raise RuntimeError("No INTEL_OPENVINO_DIR env. variable")

    subprocess.call([
        "python3", open_vino + "/deployment_tools/model_optimizer/mo_tf.py",
        # "--log_level=DEBUG",
        "--input_model", "./test.frozen.pb",
        "--input", "in1,in2",
        "--input_shape", "[1,5,64],[1,5,1]",
        "--data_type", "FP16",
        "-o", "./"])


def run_on_vpu():
    # source INTEL_OPENVINO_DIR/bin/setupvars.sh
    from openvino.inference_engine import IENetwork, IEPlugin

    plugin = IEPlugin(device="MYRIAD", plugin_dirs=None)
    plugin.set_config({
        'LOG_LEVEL': 'LOG_DEBUG',
        'VPU_PLATFORM': 'VPU_2480'
    })

    net = IENetwork(
        model="./test.frozen.xml",
        weights="./test.frozen.bin"
    )
    plugin.load(network=net, config={
        "VPU_LOG_LEVEL": "LOG_DEBUG",
        'VPU_FORCE_RESET': "NO"
    })

    # infer ....


if __name__ == '__main__':
    graph_and_freeze()
    translate()
    run_on_vpu()

And the error I receive is:

Model Optimizer version:        2019.1.0-341-gc9b66a2

[ SUCCESS ] Generated IR model.
[ SUCCESS ] XML file: /code/utils/./test.frozen.xml
[ SUCCESS ] BIN file: /code/utils/./test.frozen.bin
[ SUCCESS ] Total execution time: 2.13 seconds. 
Traceback (most recent call last):
  File "test_mat_mul.py", line 121, in <module>
    run_on_vpu()
  File "test_mat_mul.py", line 95, in run_on_vpu
    weights="./test.frozen.bin"
  File "ie_api.pyx", line 271, in openvino.inference_engine.ie_api.IENetwork.__cinit__
RuntimeError: Error reading network: in Layer out2: trying to connect an edge to non existing output port: 19.2

[shubha-ramani]

Dear @konradNowicki
Model Optimizer does support MatMul, and has done so since Day 1 of OpenVino. The below documentation shows that MatMul in Tensorflow exactly maps to the FullyConnected layer in IE.
https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Supported_Frameworks_Layers.html

Here is more information on the FullyConnected layer:
https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_IRLayersCatalogSpec.html#FullyConnected

In order to diagnose your issue further, I would need your 1) model 2) the mo command you used to generate IR and 3) a simple short version of your inference script (which you gave above).

Thanks,

Shubha

[shubha-ramani]

Dear @konradNowicki
Never mind. I see that 1) 2) and 3) I asked for are in your code already above. So you have already given me everything you need to give me. I also noticed that you are trying to run this on a VPU. And FullyConnected is definitely supported on VPU according to this document:
https://docs.openvinotoolkit.org/latest/_docs_IE_DG_supported_plugins_Supported_Devices.html

So I believe this must be a bug in the Myriad PlugIn. Thanks for your code above. I will reproduce it and file a bug against the Myriad Plugin.

Sorry for the trouble and thanks for your patience !

Shubha

[koodzi]

Dear @shubha-ramani

Matmul is supported but model optimizer might have a bug because I accidentally found that if I remove last reshape then XML is generated with without matmul and model loads to VPU.

Edit:
No, model loads but there is no matmul operation, so the reshape just exposed the bug.

        out2 = tf.matmul(out1, in1, name="out2") 
        # out2 = tf.reshape(out2, [-1, out2.shape[-1]], name="out2") 

Thanks
Konrad

[shubha-ramani]

Dearest @konradNowicki,
Actually I think it's the Myriad Plugin with the bug, not the Model Optimizer. But thanks for helping us pinpoint the issue - I will post here regarding the status.

Thanks,

Shubha

[koodzi]

I have changed the device to CPU and the error is the same.

    plugin = IEPlugin(device="CPU", plugin_dirs=None)
    net = IENetwork(
        model="./test.frozen.xml",
        weights="./test.frozen.bin"
    )
    plugin.load(network=net)

RuntimeError: Error reading network: in Layer out2: trying to connect an edge to non existing output port: 19.2

[shubha-ramani]

Dear @konradNowicki
Thanks for the additional experiment (and data point). I'm about to reproduce this myself. it looks like a bug because your model is trivially simple.

Shubha

[shubha-ramani]

Dear @konradNowicki
Just wanted to let you know that I have filed a bug on your behalf. I reproduced your issue. Thanks again for your efforts.

Stay tuned and thanks for using OpenVino !

Shubha

[studiouspan]

@shubha-ramani @konradNowicki Hello, I have the same issues： [ ERROR ] Error reading network: in Layer detector/yolo-v3-tiny/pool2/MaxPool: trying to connect an edge to non existing output port: 2.1

1 3 416 416 1 3 416 416 1 16 416 416 1 16 416 416 1 16 208 208

[ralflizard]

Hello @shubha-ramani, I'm running into a similar issue. Any insights would be appreciated.

I'm trying to load the BERT extract-features model but getting the same error.

RuntimeError: Error reading network: in Layer bert/embeddings/MatMul: trying to connect an edge to non existing output port: 13.4

This is how I'm running the model optimizer command with a saved_model folder:

mo_tf.py --saved_model_dir saved_model\1560462444 --saved_model_tags serve --output_dir bert_mo --input_shape [1],[1,128],[1,128],[1,128] --input unique_ids_1,input_ids_1,input_mask_1,input_type_ids_1 --output bert/encoder/Reshape_13,bert/encoder/Reshape_12,bert/encoder/Reshape_11,bert/encoder/Reshape_10,unique_ids_1 --log_level=WARNING 2>output.txt
Model Optimizer arguments:
Common parameters:
        - Path to the Input Model:      None
        - Path for generated IR:        C:\Users\xxxxxxx\Projects\BERT\bert\bert_mo
        - IR output name:       saved_model
        - Log level:    WARNING
        - Batch:        Not specified, inherited from the model
        - Input layers:         unique_ids_1,input_ids_1,input_mask_1,input_type_ids_1
        - Output layers:        bert/encoder/Reshape_13,bert/encoder/Reshape_12,bert/encoder/Reshape_11,bert/encoder/Reshape_10,unique_ids_1
        - Input shapes:         [1],[1,128],[1,128],[1,128]
        - Mean values:  Not specified
        - Scale values:         Not specified
        - Scale factor:         Not specified
        - Precision of IR:      FP32
        - Enable fusing:        True
        - Enable grouped convolutions fusing:   True
        - Move mean values to preprocess section:       False
        - Reverse input channels:       False
TensorFlow specific parameters:
        - Input model in text protobuf format:  False
        - Path to model dump for TensorBoard:   None
        - List of shared libraries with TensorFlow custom layers implementation:        None
        - Update the configuration file with input/output node names:   None
        - Use configuration file used to generate the model with Object Detection API:  None
        - Operations to offload:        None
        - Patterns to offload:  None
        - Use the config file:  C:\Program Files (x86)\IntelSWTools\openvino\deployment_tools\model_optimizer\extensions\front\tf\ssd_toolbox_detection_output.json
Model Optimizer version:        2019.1.1-83-g28dfbfd

[ SUCCESS ] Generated IR model.
[ SUCCESS ] XML file: C:\Users\xxxxxx\Projects\BERT\bert\bert_mo\saved_model.xml
[ SUCCESS ] BIN file: C:\Users\xxxxxx\Projects\BERT\bert\bert_mo\saved_model.bin
[ SUCCESS ] Total execution time: 58.76 seconds.

output.txt

Here are the .xml and .mapping created by mo.py
bert_mo.zip

Then just trying to load them with:

from openvino.inference_engine import IENetwork, IEPlugin

plugin = IEPlugin(device="MYRIAD")
net = IENetwork(model="./bert_mo/saved_model.xml", weights="./bert_mo/saved_model.bin")
exec_net=plugin.load(network=net, num_requests=2)

[lazarevevgeny]

The issue has been fixed. Please, verify with the latest OpenVINO from github.