MulticlassNms/MatrixNms: transformations and CPU implementation (open…

…vinotoolkit#6653) * init version, need revise: opset7 * add convert testcase * multiclass_nms support spec * init version * matrixnms support spec * init support for matrix_nms * impl matirx_nms * implemented multiclass_nms reference. TODO: more test cases. * support dynamic shape in test * update to spec 0611 * update to spec 0611 * fixes. * fix: now sort by class_id and score work. * fix clang check error * more test cases verified. * fixes in ref impl. * attribute nms_eta works * test cross_batch and output_type i32. * enable multiclass-nms cpu plugin fallback ngraph * keep topk typo * enable matrix-nms cpu plugin fallback ngraph * support sort_result_across_batch * Add matrix_nms unit test * Add cross batch test cases * fix typo * move multiclass to opset8 * move matrixnms to opset8 * Reference implementations for MulticlassNms and MatrixNms ops * fix name conflict * remove unused var sort_result_across_batch default set to false * avoid float overflow * fix clang check error * info for mac fail * change testcase due to unstable sort * nms add 'normalized' attribute * multiclass cpu test support 'normalized' * nms add 'normalized' attribute * fixes: 1. normalized support. 2. sort by score before keep_top_k inside a batch. * fixes: 1. normalized support. 2. sort by score before keep_top_k inside a batch. * fix sort order in matrix_nms * fix review comments * add matrix_nms MKLDNN extension layer * parallel in matirx nms * separate filtered_box * separate class_nms result * parallel in class * parallel in batch * partial new nms * partial remove useless function * debug & fix * debug in indexing * fix test cases * remove logging * fix code-style * fix typo * add matrix_nms extension * nms python api * remove unused testcases * refactor transformation * transform dynamic shape to static shape * Update inference-engine/src/transformations/include/ngraph_ops/nms_static_shape_ie.hpp Co-authored-by: Ilya Churaev <[email protected]> * remove register_pass call * [MKLDNN]migrate matrix_nms to MKLDNNNode * bug fix in matrix_nms * padding on matrix_nms * remove logging * test case refine * merged transform_matrix_nms branch * refine matrixnms testcase * multiclass nms cpu plugin implement for static shape, rebased on Reference implementations PR * rebase to new multi-classs transform provided by lc * Name style algin with matrix-nms * static shape padding style to batch inside,new unit test method, real classnum shape * fix format * fix ci error * multi-class NMS modification based on PR reviewer opinion: code format, copyright, delete unused include and funciton way * explicit template instantiation due to mac ci fail * Yi3/fix review (#16) * fix coding style * use parallel_for2d * fix ci fail * unify 'copyright 2021' * mkldnn_multiclass_nms node update based on PR review (#17) * [MKLDNN] apply suggestion for matrix_nms (#18) * fix bug * apply review comments * apply review comments * apply review comments * apply review comments * skip only Nms test, not MatrixNms MulticlassNms test Co-authored-by: Zhang Yi3 <[email protected]> Co-authored-by: jialipen <[email protected]> Co-authored-by: mangguo <[email protected]> Co-authored-by: Ilya Churaev <[email protected]> Co-authored-by: liubo-intel <[email protected]>
luo-cheng2021 · Jul 30, 2021 · 86bb056 · 86bb056
1 parent c0c2f2d
commit 86bb056
Show file tree

Hide file tree

Showing 31 changed files with 2,325 additions and 6 deletions.
diff --git a/inference-engine/src/inference_engine/cnn_network_ngraph_impl.cpp b/inference-engine/src/inference_engine/cnn_network_ngraph_impl.cpp
@@ -35,6 +35,9 @@
 
 #include <transformations/low_precision/disable_convert_constant_folding_on_const_path.hpp>
 
+#include <transformations/op_conversions/convert_multiclass_nms_to_multiclass_nms_ie.hpp>
+#include <transformations/op_conversions/convert_matrix_nms_to_matrix_nms_ie.hpp>
+
 #include "ie_ngraph_utils.hpp"
 #include "exec_graph_info.hpp"
 #include "ie_itt.hpp"
@@ -389,6 +392,8 @@ CNNNetworkNGraphImpl::reshape(const std::map<std::string, ngraph::PartialShape>&
                 ::ngraph::pass::Manager manager;
                 // resolves dynamism by replacing dynamic operation with static version
                 manager.register_pass<::ngraph::pass::ConvertNMS5ToLegacyMatcher>(false);
+                manager.register_pass<::ngraph::pass::ConvertMulticlassNmsToMulticlassNmsIE>();
+                manager.register_pass<::ngraph::pass::ConvertMatrixNmsToMatrixNmsIE>();
                 manager.register_pass<::ngraph::pass::DisableConvertConstantFoldingOnConstPath>();
                 manager.register_pass<::ngraph::pass::ConstantFolding>();
                 // OneHotToLegacy changes output precision

diff --git a/inference-engine/src/mkldnn_plugin/cpu_types.h b/inference-engine/src/mkldnn_plugin/cpu_types.h
@@ -86,7 +86,9 @@ enum Type {
     ExperimentalDetectronPriorGridGenerator,
     ExperimentalDetectronGenerateProposalsSingleImage,
     ExtractImagePatches,
-    NonMaxSuppression
+    NonMaxSuppression,
+    MatrixNms,
+    MulticlassNms
 };
 
 enum Algorithm {

diff --git a/inference-engine/src/mkldnn_plugin/mkldnn_node.cpp b/inference-engine/src/mkldnn_plugin/mkldnn_node.cpp
@@ -225,7 +225,9 @@ static const InferenceEngine::details::caseless_unordered_map<std::string, Type>
         { "ExperimentalDetectronPriorGridGenerator", ExperimentalDetectronPriorGridGenerator},
         { "ExperimentalDetectronGenerateProposalsSingleImage", ExperimentalDetectronGenerateProposalsSingleImage},
         { "ExtractImagePatches", ExtractImagePatches},
-        { "NonMaxSuppressionIEInternal", NonMaxSuppression}
+        { "NonMaxSuppressionIEInternal", NonMaxSuppression},
+        { "MatrixNms", MatrixNms},
+        { "MulticlassNms", MulticlassNms}
 };
 
 Type TypeFromName(const std::string type) {

diff --git a/inference-engine/src/mkldnn_plugin/mkldnn_node.h b/inference-engine/src/mkldnn_plugin/mkldnn_node.h
@@ -194,6 +194,10 @@ static std::string NameFromType(Type type) {
             return "ExtractImagePatches";
         case NonMaxSuppression:
             return "NonMaxSuppression";
+        case MatrixNms:
+            return "MatrixNms";
+        case MulticlassNms:
+            return "MulticlassNms";
         default:
             return "Unknown";
     }

diff --git a/inference-engine/src/mkldnn_plugin/mkldnn_plugin.cpp b/inference-engine/src/mkldnn_plugin/mkldnn_plugin.cpp
@@ -57,6 +57,8 @@
 #include <transformations/op_conversions/simplify_ctc_greedy_decoder_seq_len.hpp>
 #include <transformations/op_conversions/convert_previous_nms_to_nms_5.hpp>
 #include <transformations/op_conversions/convert_nms_to_nms_ie_internal.hpp>
+#include <transformations/op_conversions/convert_multiclass_nms_to_multiclass_nms_ie.hpp>
+#include <transformations/op_conversions/convert_matrix_nms_to_matrix_nms_ie.hpp>
 #include <transformations/op_conversions/convert_deformable_conv_v8_to_v1.hpp>
 #include <transformations/smart_reshape/matmul_sr.hpp>
 #include <transformations/convert_precision.hpp>
@@ -168,6 +170,8 @@ static void Transformation(CNNNetwork& clonedNetwork, const Config& conf) {
     manager.register_pass<ngraph::pass::ConvertNMS3ToNMS5>();
     manager.register_pass<ngraph::pass::ConvertNMS4ToNMS5>();
     manager.register_pass<ngraph::pass::ConvertNMSToNMSIEInternal>();
+    manager.register_pass<ngraph::pass::ConvertMulticlassNmsToMulticlassNmsIE>();
+    manager.register_pass<ngraph::pass::ConvertMatrixNmsToMatrixNmsIE>();
     manager.register_pass<ngraph::pass::TransposeMatMul>();
     manager.register_pass<ngraph::pass::ConstantFolding>();