Model Optimizer does not support array dimension value of -1

[stefan-andritoiu]

Received this error, when running model_optimizer on a frozen Tensorflow graph.
[ ERROR ] Shape [ -1 224 224 3] is not fully defined for output 0 of "input". Use --input_shape with positive integers to override model input shapes.

From the numpy.reshape() documentation (https://docs.scipy.org/doc/numpy/reference/generated/numpy.reshape.html):
"One shape dimension can be -1. In this case, the value is inferred from the length of the array and remaining dimensions.

The error appears because the model_optimizer requires all dimensions to be positive. And yes, if you use --input_shape [1,3,224,224] as command-line parameter, it works perfectly fine.

The issue here is that it's common practice in the entire "ML in python" field to have the batch size -1, as you don't want to constrict the model to a fixed batch size, but inferred at run time.
And almost every Tensorflow model uses -1 as batch size. So this will be an error that will pop-up every time someone just downloads a model from the internet and tries to optimize it.

Are there any plans of adding support for this type of "run-time inferred dimension" ?

[shubha-ramani]

stefan it is under discussion. I have informed the Model Optimizer development team of your request.

[lazarevevgeny]

@stefan-andritoiu the current workflow is that you have to specify positive values for all dimensions of the model input during the model conversion. Then you can change the batch size (and the spatial dimensions of the image also) at runtime using the Inference Engine API method InferenceEngine::CNNNetwork::reshape.

There are no plans to support dynamic shapes in the Model Optimizer.

[elvin-n]

We cannot have undefined shape in IR because model eventually must be loaded to IE plugin and all shapes must be defined. Even for dynamic batch that I will mention later, you must have all values defined.

Thus, IR have all values predefined and there might be a question "what should I do if I want to change input shape"? The answer on this question - shape propagation feature in Inference Engine or explicit call of setBatchSize for ICNNNetwork. There are two different functions functionality of which are overlapped and there might be a change in API in this regard to leave only one approach or to address issues with ICNNNetwork::setBatchSize

After loading model from IR and before loading model to plugin, you can change input shape or batch size as a partial use case of changing shape.
Here is an example:

SizeVector input_shape;
std::tie(input_name, input_shape) = *input_shapes.begin();
input_shape[0] = batch_size;
input_shapes[input_name] = input_shape;
network.reshape(input_shapes);
ExecutableNetwork executable_network = plugin.LoadNetwork(network, {});

Function ICNNNetwork::reshape might fail if layer cannot do shape propagation. For example if network has fully connected layers and if you want to change width ot heihgt.

This approach should be used for changing of batch size for network. It works for all cases.

As an alternative to shape propagation - there is a function ICNNNetwork::setBatchSize. Call of this function is simple, but it has many limitation. This function does not use for now shape inference and assumes that batch dimension is always the first. And it just change this value for all layers. It works well for many networks but there are a lot of networks having layers which do not have dedicated dimension for batch. For example Detection Output layer or Proposal. In this case there might be unpredictable behaviour till the application crash.
Limitations are described in the comment for ICNNNetwork.
Use of this function is simple but we recommend to use shape inference instead even for changing of batch.

Previous two examples described use cases of changing shapes before loading network to plugin. After loading network to plugin, you cannot change anything in the inference configuration. All actions on ICNNNetwork will not be applied to already loaded network and their representatives ExecutableNetwork/InferRequest.

There is one more use case - dynamic batch size which can be changed from inference to infernece. Even in this case the batch size must be defined on moment of loading network to plugin. This batch size means the maximum batch which you cannot exceed for dynamic batch for certain ExecutableNetwork. The second prerequisite to turn on this dynamic batch - send config key PluginConfigParams::KEY_DYN_BATCH_ENABLED to YES.
And third prerequisite - you can use IInferRequest::SetBatch() to define of batch size for next infer calls
This function has same limitations for now as ICNNNetwork::setBatchSize

[lazarevevgeny]

@stefan-andritoiu could you close the ticket if you don't have any more comments?

[jlygit]

@elvin-nnov

I change two input shapes and it doesn't work for the second input when the IR has two input nodes. why?

the code is

and the printf

[mrinmayk]

I am having similar errors; someone please solve issue with multiple input node issue.

[lazarevevgeny]

Closing the ticket since there are no plans to support dynamic input shape.