diff --git a/docs/IE_DG/ShapeInference.md b/docs/IE_DG/ShapeInference.md index a7cdddb784d676..ea86911ff397e0 100644 --- a/docs/IE_DG/ShapeInference.md +++ b/docs/IE_DG/ShapeInference.md @@ -1,6 +1,36 @@ Using Shape Inference {#openvino_docs_IE_DG_ShapeInference} ========================================== +OpenVINO™ provides the following methods for runtime model reshaping: + +* **Set a new input shape** with the `InferenceEngine::CNNNetwork::reshape` method.
+ The `InferenceEngine::CNNNetwork::reshape` method updates input shapes and propagates them down to the outputs of the model through all intermediate layers. + +> **NOTES**: +> - Starting with the 2021.1 release, the Model Optimizer converts topologies keeping shape-calculating sub-graphs by default, which enables correct shape propagation during reshaping in most cases. +> - Older versions of IRs are not guaranteed to reshape successfully. Please regenerate them with the Model Optimizer of the latest version of OpenVINO™.
+> - If an ONNX model does not have a fully defined input shape and the model was imported with the ONNX importer, reshape the model before loading it to the plugin. + +* **Set a new batch dimension value** with the `InferenceEngine::CNNNetwork::setBatchSize` method.
+ The meaning of a model batch may vary depending on the model design. + This method does not deduce batch placement for inputs from the model architecture. + It assumes that the batch is placed at the zero index in the shape for all inputs and uses the `InferenceEngine::CNNNetwork::reshape` method to propagate updated shapes through the model. + + The method transforms the model before a new shape propagation to relax a hard-coded batch dimension in the model, if any. + + Use `InferenceEngine::CNNNetwork::reshape` instead of `InferenceEngine::CNNNetwork::setBatchSize` to set new input shapes for the model in case the model has: + * Multiple inputs with different zero-index dimension meanings + * Input without a batch dimension + * 0D, 1D, or 3D shape + + The `InferenceEngine::CNNNetwork::setBatchSize` method is a high-level API method that wraps the `InferenceEngine::CNNNetwork::reshape` method call and works for trivial models from the batch placement standpoint. + Use `InferenceEngine::CNNNetwork::reshape` for other models. + + Using the `InferenceEngine::CNNNetwork::setBatchSize` method for models with a non-zero index batch placement or for models with inputs that do not have a batch dimension may lead to undefined behaviour. + +You can change input shapes multiple times using the `InferenceEngine::CNNNetwork::reshape` and `InferenceEngine::CNNNetwork::setBatchSize` methods in any order. +If a model has a hard-coded batch dimension, use `InferenceEngine::CNNNetwork::setBatchSize` first to change the batch, then call `InferenceEngine::CNNNetwork::reshape` to update other dimensions, if needed. + Inference Engine takes three kinds of a model description as an input, which are converted into an `InferenceEngine::CNNNetwork` object: 1. [Intermediate Representation (IR)](../MO_DG/IR_and_opsets.md) through `InferenceEngine::Core::ReadNetwork` 2. [ONNX model](../IE_DG/OnnxImporterTutorial.md) through `InferenceEngine::Core::ReadNetwork` @@ -23,33 +53,7 @@ for (const auto & parameter : parameters) { To feed input data of a shape that is different from the model input shape, reshape the model first. -OpenVINO™ provides the following methods for runtime model reshaping: - -* **Set a new input shape** with the `InferenceEngine::CNNNetwork::reshape` method.
- The `InferenceEngine::CNNNetwork::reshape` method updates input shapes and propagates them down to the outputs of the model through all intermediate layers. - You can reshape a model multiple times like in this application scheme: - ``` - ReadNetwork -> reshape(input_1_shape) -> LoadNetwork -> infer(input_1) - \ - -> reshape(input_2_shape) -> LoadNetwork -> infer(input_2) - ``` - > **NOTES**: - > - Starting with the 2021.1 release, the Model Optimizer converts topologies keeping shape-calculating sub-graphs by default, which enables correct shape propagation during reshaping. - > - Older versions of IRs are not guaranteed to reshape successfully. Please regenerate them with the Model Optimizer of the latest version of OpenVINO™.
- > - If an ONNX model does not have a fully defined input shape and the model was imported with the ONNX importer, reshape the model before loading it to the plugin. -* **Set a new batch dimension value** with the `InferenceEngine::CNNNetwork::setBatchSize` method.
- The meaning of a model batch may vary depending on the model design. - The `InferenceEngine::CNNNetwork::setBatchSize` method deduces the index of a batch dimension based only on the input rank. - This method does not work for models with a non-zero index batch placement or models with inputs without a batch dimension. - The batch-setting algorithm does not involve the shape inference mechanism. - Batch of input and output shapes for all layers is set to a new batch value without layer validation. - It may cause both positive and negative side effects. - Due to the limitations described above, the current method is not recommended to use. - If you need to set a new batch size for the model, use the `CNNNetwork::reshape` method instead. - -Do not use runtime reshaping methods simultaneously, especially do not call the `CNNNetwork::reshape` method after you use `InferenceEngine::CNNNetwork::setBatchSize`. -The `InferenceEngine::CNNNetwork::setBatchSize` method causes irreversible conversion of the internal model representation into the legacy model representation. -The method does not use nGraph for shape inference which leads to reduced reshape opportunities and may affect the performance of the model. +Once the input shape of `InferenceEngine::CNNNetwork` is set, call the `InferenceEngine::Core::LoadNetwork` method to get an `InferenceEngine::ExecutableNetwork` object for inference with updated shapes. There are other approaches to reshape the model during the stage of IR generation or [nGraph::Function creation](../nGraph_DG/build_function.md). @@ -62,8 +66,8 @@ Shape collision during shape propagation may be a sign that a new shape does not Changing the model input shape may result in intermediate operations shape collision. Examples of such operations: -- `Reshape` operation with a hard-coded output shape value -- `MatMul` operation with the `Const` second input cannot be resized by spatial dimensions due to operation semantics +- [`Reshape` operation](../ops/shape/Reshape_1.md) with a hard-coded output shape value +- [`MatMul` operation](../ops/matrix/MatMul_1.md) with the `Const` second input cannot be resized by spatial dimensions due to operation semantics Model structure and logic should not change significantly after model reshaping. - The Global Pooling operation is commonly used to reduce output feature map of classification models output.