MatrixNMS-8 spec. (#5948)

* Specification of a new operator called MatrixNMS. This Op functionally equivalent to NonMaxSuppression-5, with an non max suppression algorithm called matrix_nms, and perform more post-processing phases, and lay out the detection outputs in the way of PaddlePaddle detection. * Update docs/ops/sort/MatrixNMS_8.md Co-authored-by: Anastasiya Ageeva <[email protected]> * Update docs/ops/sort/MatrixNMS_8.md Co-authored-by: Anastasiya Ageeva <[email protected]> * Update docs/ops/sort/MatrixNMS_8.md Co-authored-by: Anastasiya Ageeva <[email protected]> * Update docs/ops/sort/MatrixNMS_8.md Co-authored-by: Anastasiya Ageeva <[email protected]> * Update docs/ops/sort/MatrixNMS_8.md Co-authored-by: Anastasiya Ageeva <[email protected]> * Update docs/ops/sort/MatrixNMS_8.md Co-authored-by: Anastasiya Ageeva <[email protected]> * Update docs/ops/sort/MatrixNMS_8.md Co-authored-by: Anastasiya Ageeva <[email protected]> * Update docs/ops/sort/MatrixNMS_8.md Co-authored-by: Anastasiya Ageeva <[email protected]> * Update MatrixNMS_8.md * Update MatrixNMS_8.md Co-authored-by: Anastasiya Ageeva <[email protected]>
openvinotoolkit · Jun 18, 2021 · ac01777 · ac01777
1 parent 020af3d
commit ac01777
Show file tree

Hide file tree

Showing 2 changed files with 169 additions and 0 deletions.
diff --git a/docs/doxygen/ie_docs.xml b/docs/doxygen/ie_docs.xml
@@ -189,6 +189,7 @@ limitations under the License.
                 <tab type="user" title="MVN-1" url="@ref openvino_docs_ops_normalization_MVN_1"/>
                 <tab type="user" title="MVN-6" url="@ref openvino_docs_ops_normalization_MVN_6"/>
                 <tab type="user" title="MatMul-1" url="@ref openvino_docs_ops_matrix_MatMul_1"/>
+                <tab type="user" title="MatrixNonMaxSuppression-8" url="@ref openvino_docs_ops_sort_MatrixNonMaxSuppression_8"/>
                 <tab type="user" title="MaxPool-1" url="@ref openvino_docs_ops_pooling_MaxPool_1"/>
                 <tab type="user" title="Maximum-1" url="@ref openvino_docs_ops_arithmetic_Maximum_1"/>
                 <tab type="user" title="Minimum-1" url="@ref openvino_docs_ops_arithmetic_Minimum_1"/>

diff --git a/docs/ops/sort/MatrixNMS_8.md b/docs/ops/sort/MatrixNMS_8.md
@@ -0,0 +1,168 @@
+##  MatrixNonMaxSuppression<a name="MatrixNonMaxSuppression"></a> {#openvino_docs_ops_sort_MatrixNms_8}
+
+**Versioned name**: *MatrixNonMaxSuppression-8*
+
+**Category**: *Sorting and maximization*
+
+**Short description**: *MatrixNonMaxSuppression* performs matrix non-maximum suppression (NMS) of the boxes with predicted scores.
+
+**Detailed description**: The operation performs the following:
+
+1. Selects candidate bounding boxes with scores higher than `score_threshold`.
+2. For each class, selects at most `nms_top_k` candidate boxes.
+3. Decays scores of the candidate boxes according to the Matrix NMS algorithm [Wang et al](https://arxiv.org/abs/2003.10152.pdf). This algorithm is applied independently to each class and each batch element. Boxes of `background_class` are skipped and thus eliminated during the process.
+4. Selects boxes with the decayed scores higher than `post_threshold`, and selects at most `keep_top_k` scoring candidate boxes per batch element.
+
+The Matrix NMS algorithm is described below:
+1.  Sort descending the candidate boxes by score, and compute `n*n` pairwise IOU (IntersectionOverUnion) matrix `X` for the top `n` boxes. Suppose `n` is the number of candidate boxes.
+2.  Set the lower triangle and diagonal of `X` to 0. Therefore get the upper triangular matrix `X`.
+3.  Take the column-wise max of `X` to compute a vector `K` of maximum IOU for each candidate box.
+4.  Repeat element value of `K` along axis 1. Suppose this gets a matrix `X_cmax`.
+5.  Compute the decay factor: `decay_factor = exp((X_cmax**2 - X**2) * gaussian_sigma)` if `decay_function` is `guassian`, else `decay_factor = (1 - X) / (1 - X_cmax)`.
+6.  Take the column-wise min of `decay_factor`, and element-wise multiply with scores to decay them.
+
+**Attributes**:
+
+* *sort_result*
+
+  * **Description**: *sort_result* specifies the order of output elements.
+  * **Range of values**: `class`, `score`, `none`
+    * *class* - sort selected boxes by class id (ascending).
+    * *score* - sort selected boxes by score (descending).
+    * *none* - do not guarantee the order.
+  * **Type**: `string`
+  * **Default value**: `none`
+  * **Required**: *No*
+
+* *sort_result_across_batch*
+
+  * **Description**: *sort_result_across_batch* is a flag that specifies whenever it is necessary to sort selected boxes across batches or not.
+  * **Range of values**: true or false
+    * *true* - sort selected boxes across batches.
+    * *false* - do not sort selected boxes across batches (boxes are sorted per batch element).
+  * **Type**: boolean
+  * **Default value**: false
+  * **Required**: *No*
+
+* *output_type*
+
+  * **Description**: the tensor type of outputs `selected_indices` and `valid_outputs`.
+  * **Range of values**: `i64` or `i32`
+  * **Type**: `string`
+  * **Default value**: `i64`
+  * **Required**: *No*
+
+* *score_threshold*
+
+  * **Description**: minimum score to consider box for the processing.
+  * **Range of values**: a floating-point number
+  * **Type**: `float`
+  * **Default value**: `0`
+  * **Required**: *No*
+
+* *nms_top_k*
+
+  * **Description**: maximum number of boxes to be selected per class.
+  * **Range of values**: an integer
+  * **Type**: `int`
+  * **Default value**: `-1` meaning to keep all boxes
+  * **Required**: *No*
+
+* *keep_top_k*
+
+  * **Description**: maximum number of boxes to be selected per batch element.
+  * **Range of values**: an integer
+  * **Type**: `int`
+  * **Default value**: `-1` meaning to keep all boxes
+  * **Required**: *No*
+
+* *background_class*
+
+  * **Description**: the background class id.
+  * **Range of values**: an integer
+  * **Type**: `int`
+  * **Default value**: `-1` meaning to keep all classes
+  * **Required**: *No*
+
+* *decay_function*
+
+  * **Description**: decay function used to decay scores.
+  * **Range of values**: `gaussian`, `linear`
+  * **Type**: `string`
+  * **Default value**: `linear`
+  * **Required**: *No*
+
+* *gaussian_sigma*
+
+  * **Description**: gaussian_sigma parameter for gaussian decay_function.
+  * **Range of values**: a floating-point number
+  * **Type**: `float`
+  * **Default value**: `2.0`
+  * **Required**: *No*
+
+* *post_threshold*
+
+  * **Description**: threshold to filter out boxes with low confidence score after decaying.
+  * **Range of values**: a floating-point number
+  * **Type**: `float`
+  * **Default value**: `0`
+  * **Required**: *No*
+
+**Inputs**:
+
+*   **1**: `boxes` - tensor of type *T* and shape `[num_batches, num_boxes, 4]` with box coordinates. The box cooridnates are layout as `[xmin, ymin, xmax, ymax]`. **Required.**
+
+*   **2**: `scores` - tensor of type *T* and shape `[num_batches, num_classes, num_boxes]` with box scores. **Required.**
+
+**Outputs**:
+
+*   **1**: `selected_outputs` - tensor of type *T_THRESHOLDS* and shape `[number of selected boxes, 6]` containing the selected boxes with score and class as tuples `[class_id, box_score, xmin, ymin, xmax, ymax]`.
+
+*   **2**: `selected_indices` - tensor of type *T_IND* and shape `[number of selected boxes, 1]` the selected indices in the flattened input `boxes`, which are absolute values cross batches. Therefore possible valid values are in the range `[0, num_batches * num_boxes - 1]`.
+
+*   **3**: `selected_num` - 1D tensor of type *T_IND* and shape `[num_batches]` representing the number of selected boxes for each batch element.
+
+When there is no box selected, `selected_num` is filled with `0`. `selected_outputs` is an empty tensor of shape `[0, 6]`, and `selected_indices` is an empty tensor of shape `[0, 1]`.
+
+**Types**
+
+* *T*: floating point type.
+
+* *T_MAX_BOXES*: integer type.
+
+* *T_THRESHOLDS*: floating point type.
+
+* *T_IND*: `int64` or `int32`.
+
+**Example**
+
+```xml
+<layer ... type="MatrixNonMaxSuppression" ... >
+    <data decay_function="guassian" sort_result="score" output_type="i64"/>
+    <input>
+        <port id="0">
+            <dim>3</dim>
+            <dim>100</dim>
+            <dim>4</dim>
+        </port>
+        <port id="1">
+            <dim>3</dim>
+            <dim>5</dim>
+            <dim>100</dim>
+        </port>
+    </input>
+    <output>
+        <port id="5" precision="FP32">
+            <dim>-1</dim> <!-- "-1" means a undefined dimension calculated during the model inference -->
+            <dim>6</dim>
+        </port>
+        <port id="6" precision="I64">
+            <dim>-1</dim>
+            <dim>1</dim>         
+        </port>
+        <port id="7" precision="I64">
+            <dim>3</dim>
+        </port>
+    </output>
+</layer>
+```