Enable NonMaxSuppression-5 operation and various transformations (ope…

…nvinotoolkit#2450) * Some code style fixes. * Started to write the method v5::NonMaxSuppression::evaluate(). * Started to write nGraph reference implementation of NMS-5. * Some additions. * Written preprocessing of boxes data. * Started to write the function non_max_suppression() that calculates NonMaxSuppression-5 operation. * Written postprocessing of the evaluate(). * Small fixes. * Small fix. * Added include for ngraph/shape.hpp. * Written the function intersectionOverUnion. * Some additions. * Small fix. * Continued to write the function non_max_suppression(). * Small fixes. * Small fixes. * Some changes. * Some additions. * Some replacements size_t by int64_t. * Added casts to float in the construction of selected_score variable. * Code style fixes. * Written draft of NMS-5 nGraph reference implementation. * Code style fixes. * Started to write tests for void op::v5::NonMaxSuppression::validate_and_infer_types(). * Added tests for scalars/nonscalars. * Fixes in the test type_prop.nms_v5_output_shape. * Fixes in tests nms_v5_output_shape_2 and nms_v5_output_shape. * Written tests for validate_and_infer_types() of NMS-5. * Code style fixes. * Now NMS-5 evaluate() can have outputs with calculated shapes. * Small fixes. * Corrected tests for NMS-5 validate_and_infer_type(). * Code style fixes. * Started to write inner version of NMS-5 with static output shapes. * Written draft of the inner operation NonMaxSuppressionIE3. * Started to write conversion of op::v5::NonMaxSuppression into NonMaxSuppressionIE3. * Small changes. * Some additions. * Small fixes. * Fixed typo. * Fixed typos. * Written draft of the transformation ConvertNMS5ToLegacyMatcher that converts ngraph::opset5::NonMaxSuppression into op::NonMaxSuppressionIE3. * Written header file for the transformations from NMS-1, NMS-3, NMS-4 to NMS-5. * Started to write conversion of NMS-4 to NMS-5. * Added include for ngraph/opsets/opset4.hpp. * Started to write conversion of NMS-3 to NMS-5. * Small fixes. * Written draft of the conversion of NMS-3 into NMS-5. * Fixed typo. * Started to write conversion of NMS-1 to NMS-5. * Written draft of the conversion NMS-1 to NMS-5. * Started to write tests for the conversion nGraph NMS-5 to inner NMS. * Started to write the test ConvertNMS5ToNMSIEStatic. * Written tests for conversion of nGraph NMS-5 to inner NMSIE3. * Started to write tests for conversion of previous NMS to nGraph NMS-5. * Written tests for conversion of old nGraph NMS to NMS-5. * Started to write tests for opset5::NonMaxSuppression::evaluate(). * Some additions. * Small fix. * Written tests for op::v5::NonMaxSuppression::evaluate(). * Used NGRAPH_RTTI_DECLARATION for NonMaxSuppressionIE3. * Used NGRAPH_RTTI_DECLARATION for NMS-5. * All static local constants and functions for NMS-5 were moved into non-named namespace. * Some code style fixes. * Moved some file. * Small fix. * Code style fix. * Now NMS-5 supports all floating types in inputs 0 and 1. * Moved some files. * Fixed include directive in the file convert_nms_5_to_legacy.cpp with transformations NMS-1, 3, 4 -> NMS-5. * Small changes. * Deleted conversion NMS-3 -> legacy. * Small changes. * Fix in op::v5::NonMaxSuppression::evaluate: output shape [1] instead of [] in the output port 2. * Code style fixes. * Deleted conversion of NMS-4 into legacy NMS. * Deleted redundant ifs. * Added NMS-5 to Python API. * Code style fix. * Small change. * Fixed element type for constants in the conversion of NMS-5 to NMSIE3. * Deleted support of f64 in NMS-5. * Added checks for input element types for inputs #0, #1, #3, #4, #5. * Small change. * Now get_floats throws an exception for unsupported types. * Now nGraph NMS-5 supports 0D and 1D tensors in inputs #2, #3, #4, #5. * Small fix in test_non_max_suppression. * Deleted using namespace std * Fixes in test_non_max_suppression(). * Small fixes. * Added 'import PartialShape' in test_reduction.py. * Deleted creating fake inputs in the ctor of op::v5::NonMaxSuppression. * Deleted creating fake inputs in op::v5::NonMaxSuppression::clone_with_new_inputs. * Corrections in int64_t op::v5::NonMaxSuppression::max_boxes_output_from_input() const. * Corrected functions float op::v5::NonMaxSuppression::soft_nms_sigma_from_input() const, float op::v5::NonMaxSuppression::score_threshold_from_input() const, float op::v5::NonMaxSuppression::iou_threshold_from_input() const. * Small fixes. * Deleted commented code. * Fixes in nms_v5_scalar_inputs_check. * Some changes. * Small fixes. * Code style fixes. * Small changes. * Small changes. * Small fix. * Deleted commented code. * Some refactoring in ConvertNMS4ToNMS5 ctor. * Small fix. * Common part of conversions NMS-1 -> NMS-5, NMS-3 -> NMS-5, NMS-4 -> NMS-5 was moved into the separate function. * Now conversions NMS-1 -> NMS-5, NMS-3 -> NMS-5, NMS-4 -> NMS-5 support both 2 inputs, and 5 inputs. * Now transformations NMS-1 -> NMS-5, NMS-3 -> NMS-5, NMS-4 -> NMS-5 are called from 'umbrella' transformation. * Now the transformation ConvertNMS5ToLegacyMatcher supports NMS-5 with 2, 3, 4, 5, or 6 inputs. * The transformation ConvertNMS5ToLegacyMatcher was rewritten using Reshape instead of Unsqueeze. * Started to rewrite tests for the transformation ConvertNMS5ToLegacyMatcher. * Some fixes. * Small fixes. * Corrected tests for the transformation NMS-5 -> NMSIE3. * Small formatting fix. * Now methods max_boxes_output_from_input(), iou_threshold_from_input(), score_threshold_from_input(), soft_nms_sigma_from_input() of op::v5::NunMaxSuppression are public. * Started to move op::v5::NonMaxSuppression::evaluate() into ngraph/test/runtime/interpreter. * Added NMS-5 to ngraph/test/runtime/interpreter/int_executable. * Small fixes. * Code style fixes. * Written draft test nonmaxsuppression_center_point_box_format_backend in ngraph/test/backend. * Small fix. * Written draft tests of NonMaxSuppression in ngraph/test/runtime. * Some changes. * Small changes. * Disabled IE_CPU tests for NMS-5. * Deleted op_eval tests for NMS-5. * Deleted evaluate() method of NMS-5. * Now all nGraph functions in tests of the transformation NMS-5 -> NMSIE3 have one output. * Now preprocessing and postprocessing of the calculation of NMS-5 in reference implementation. * Code style fixes. * Some fixes in tests for the transformation NMS-5 -> NMSIE3. * Replaced precision i64 -> i32 for some constants in tests for the transformation NMS-5 -> NMSIE3. * Written creating CNNLayer for NMS-5. * Added creating CNNLayer for NonMaxSuppressionIE3. * some changes. * Now conversions NMS-1, NMS-3, NMS-4 -> NMS-5 and NMS-5 -> NMSIE3 generate NMS nodes with 5 inputs. * Fixed ctor in MKLDNN NonMaxSuppressionImpl: validation of number of output edges. * Added conversion of output_type for NMS-5. * Fixes in the transformation NMS5 -> NMSIE3. * Fixes in the conversion of NMS-5 to NMSIE3. * Fixes in MKLDNN NMS ctor. * Small fix. * Fixed tests for the transformation NMS5 -> NMSIE3. * Fixed tests for conversions NMS-1, NMS-3, NMS-4 -> NMS-5. * Small fixes in MKLDNN NMS ctor. * Rewritten create_layer() functions for NMS-5 and NMSIE3 as addSpecificCreator() functions. * Disabled tests for IE IR reader for NMS-4. * Deleted debug code. * Added comment about disabling tests IE_CPU.nonmaxsuppression. * Written IE IR Reader test for NMS-4. * Deleted function float_from_constant_node. * Small fixes. * Deleted functions function_from_model and construct_weights. * Small fix. * Replaced push_back with emplace_back in the conversion of NMS-5 to legacy. * Small changes. * Some fixes. * Refactored reference implementation of NMS-5. * Moved structure NMSAttributes to unnamed namespace. * Code style fixes. * Small fix.
mryzhov · Dec 15, 2020 · a4b19b4 · a4b19b4
1 parent 2a61a4e
commit a4b19b4
Show file tree

Hide file tree

Showing 36 changed files with 3,399 additions and 561 deletions.
diff --git a/inference-engine/src/legacy_api/include/legacy/ie_layers.h b/inference-engine/src/legacy_api/include/legacy/ie_layers.h
@@ -2064,6 +2064,10 @@ class INFERENCE_ENGINE_INTERNAL_CNNLAYER_CLASS(NonMaxSuppressionLayer): public C
      * classes
      */
     bool sort_result_descending = true;
+    /**
+     * @brief Output type for first and third inputs
+     */
+    std::string output_type = "I64";
     /**
      * @brief Creates a new NonMaxSuppressionLayer instance.
      */

diff --git a/inference-engine/src/legacy_api/include/legacy/ngraph_ops/nms_ie.hpp b/inference-engine/src/legacy_api/include/legacy/ngraph_ops/nms_ie.hpp
@@ -62,6 +62,15 @@ class INFERENCE_ENGINE_API_CLASS(NonMaxSuppressionIE3) : public Op {
 public:
     NGRAPH_RTTI_DECLARATION;
 
+    NonMaxSuppressionIE3(const Output<Node>& boxes,
+                         const Output<Node>& scores,
+                         const Output<Node>& max_output_boxes_per_class,
+                         const Output<Node>& iou_threshold,
+                         const Output<Node>& score_threshold,
+                         int center_point_box,
+                         bool sort_result_descending,
+                         const ngraph::element::Type& output_type = ngraph::element::i64);
+
     NonMaxSuppressionIE3(const Output<Node>& boxes,
                          const Output<Node>& scores,
                          const Output<Node>& max_output_boxes_per_class,

diff --git a/...y_api/include/legacy/transformations/convert_opset1_to_legacy/convert_nms_4_to_legacy.hpp b/...y_api/include/legacy/transformations/convert_opset1_to_legacy/convert_nms_4_to_legacy.hpp
diff --git a/...ormations/op_conversions/convert_nms3.hpp → ...et1_to_legacy/convert_nms_5_to_legacy.hpp b/...ormations/op_conversions/convert_nms3.hpp → ...et1_to_legacy/convert_nms_5_to_legacy.hpp
@@ -14,18 +14,20 @@
 namespace ngraph {
 namespace pass {
 
-class TRANSFORMATIONS_API ConvertNMS1ToNMS3;
+class INFERENCE_ENGINE_API_CLASS(ConvertNMS5ToLegacyMatcher);
 
 }  // namespace pass
 }  // namespace ngraph
 
-class ngraph::pass::ConvertNMS1ToNMS3: public ngraph::pass::GraphRewrite {
+/*
+ * Description:
+ *      Convert NMS-5 directly to inner NMS.
+ */
+
+
+class ngraph::pass::ConvertNMS5ToLegacyMatcher: public ngraph::pass::MatcherPass {
 public:
     NGRAPH_RTTI_DECLARATION;
-    ConvertNMS1ToNMS3() : GraphRewrite() {
-        convert_nms1_to_nms3();
-    }
-
-private:
-    void convert_nms1_to_nms3();
+    ConvertNMS5ToLegacyMatcher();
 };
+
diff --git a/inference-engine/src/legacy_api/src/convert_function_to_cnn_network.cpp b/inference-engine/src/legacy_api/src/convert_function_to_cnn_network.cpp
@@ -653,6 +653,83 @@ InferenceEngine::details::CNNLayerCreator::CNNLayerCreator(const std::shared_ptr
 
     });
 
+    addSpecificCreator({"NonMaxSuppressionIE3"}, [](const std::shared_ptr<::ngraph::Node>& node,
+        const std::map<std::string, std::string>& params) -> CNNLayerPtr {
+        LayerParams attrs = {node->get_friendly_name(), "NonMaxSuppression",
+            details::convertPrecision(node->get_output_element_type(0))};
+
+        auto castedLayer = ::ngraph::as_type_ptr<::ngraph::op::NonMaxSuppressionIE3>(node);
+        IE_ASSERT(castedLayer) << " Operation " << node->description() << " with name "
+            << node->get_friendly_name() << " cannot be casted to ngraph::op::NonMaxSuppressionIE3";
+
+        auto res = std::make_shared<InferenceEngine::NonMaxSuppressionLayer>(attrs);
+        res->params = params;
+
+        res->params["center_point_box"] = castedLayer->m_center_point_box ? "true" : "false";
+        res->params["sort_result_descending"] = castedLayer->m_sort_result_descending ? "true" : "false";
+
+        auto output_type = details::convertPrecision(castedLayer->m_output_type);
+        std::string output_type_str;
+        switch (output_type) {
+        case Precision::I32:
+            output_type_str = "I32";
+            break;
+        case Precision::I64:
+            output_type_str = "I64";
+            break;
+        default:
+            THROW_IE_EXCEPTION << "Unsupported output type";
+        }
+        res->params["output_type"] = output_type_str;
+
+        return res;
+    });
+
+    addSpecificCreator({"NonMaxSuppression"}, [](const std::shared_ptr<::ngraph::Node>& node,
+        const std::map<std::string, std::string>& params) -> CNNLayerPtr {
+        LayerParams attrs = {node->get_friendly_name(), "NonMaxSuppression",
+            details::convertPrecision(node->get_output_element_type(0))};
+
+        auto castedLayer = ::ngraph::as_type_ptr<::ngraph::op::v5::NonMaxSuppression>(node);
+        IE_ASSERT(castedLayer) << " Operation " << node->description() << " with name "
+            << node->get_friendly_name() << " cannot be casted to ngraph::op::v5::NonMaxSuppression";
+
+        auto res = std::make_shared<InferenceEngine::NonMaxSuppressionLayer>(attrs);
+        res->params = params;
+
+        auto box_encoding = castedLayer->get_box_encoding();
+        switch (box_encoding) {
+            case ngraph::op::v5::NonMaxSuppression::BoxEncodingType::CORNER:
+                res->params["center_point_box"] = "false";
+                break;
+            case ngraph::op::v5::NonMaxSuppression::BoxEncodingType::CENTER:
+                res->params["center_point_box"] = "true";
+                break;
+            default:
+                THROW_IE_EXCEPTION << "Unsupported box encoding for NonMaxSuppression op";
+                break;
+        }
+
+        auto output_type = details::convertPrecision(castedLayer->get_output_type());
+        std::string output_type_str;
+        switch (output_type) {
+        case Precision::I32:
+            output_type_str = "I32";
+            break;
+        case Precision::I64:
+            output_type_str = "I64";
+            break;
+        default:
+            THROW_IE_EXCEPTION << "Unsupported output type";
+        }
+        res->params["output_type"] = output_type_str;
+
+        bool sort_result_descending = castedLayer->get_sort_result_descending();
+        res->params["sort_result_descending"] = sort_result_descending ? "true" : "false";
+
+        return res;
+    });
+
     addSpecificCreator({"NonMaxSuppressionIE"}, [](const std::shared_ptr<::ngraph::Node>& node,
                                                  const std::map<std::string, std::string>& params) -> CNNLayerPtr {
         LayerParams attrs = {node->get_friendly_name(), "NonMaxSuppression", details::convertPrecision(node->get_output_element_type(0))};

diff --git a/inference-engine/src/legacy_api/src/ie_layer_validators.cpp b/inference-engine/src/legacy_api/src/ie_layer_validators.cpp
@@ -1181,6 +1181,7 @@ void NMSValidator::parseParams(CNNLayer* layer) {
 
     casted->center_point_box = layer->GetParamAsBool("center_point_box", false);
     casted->sort_result_descending = layer->GetParamAsBool("sort_result_descending", true);
+    casted->output_type = layer->GetParamAsString("output_type", "I64");
 }
 
 #define REG_LAYER_VALIDATOR_FOR_TYPE(__validator, __type) _validators[#__type] = std::make_shared<__validator>(#__type)

diff --git a/inference-engine/src/legacy_api/src/ngraph_ops/nms_ie.cpp b/inference-engine/src/legacy_api/src/ngraph_ops/nms_ie.cpp
@@ -102,7 +102,20 @@ void op::NonMaxSuppressionIE2::validate_and_infer_types() {
     set_output_type(0, nms->output(0).get_element_type(), nms->output(0).get_partial_shape());
 }
 
-NGRAPH_RTTI_DEFINITION(op::NonMaxSuppressionIE3, "NonMaxSuppressionIE", 3);
+NGRAPH_RTTI_DEFINITION(op::NonMaxSuppressionIE3, "NonMaxSuppressionIE3", 3);
+
+op::NonMaxSuppressionIE3::NonMaxSuppressionIE3(const Output<Node>& boxes,
+                                               const Output<Node>& scores,
+                                               const Output<Node>& max_output_boxes_per_class,
+                                               const Output<Node>& iou_threshold,
+                                               const Output<Node>& score_threshold,
+                                               int center_point_box,
+                                               bool sort_result_descending,
+                                               const ngraph::element::Type& output_type)
+        : Op({boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold}),
+          m_center_point_box(center_point_box), m_sort_result_descending(sort_result_descending), m_output_type(output_type) {
+    constructor_validate_and_infer_types();
+}
 
 op::NonMaxSuppressionIE3::NonMaxSuppressionIE3(const Output<Node>& boxes,
                                                const Output<Node>& scores,
@@ -139,8 +152,13 @@ static constexpr size_t max_output_boxes_per_class_port = 2;
 int64_t op::NonMaxSuppressionIE3::max_boxes_output_from_input() const {
     int64_t max_output_boxes{0};
 
+    size_t num_of_inputs = inputs().size();
+    if (num_of_inputs < 3) {
+        return 0;
+    }
+
     const auto max_output_boxes_input =
-        as_type_ptr<op::Constant>(input_value(2).get_node_shared_ptr());
+        as_type_ptr<op::Constant>(input_value(max_output_boxes_per_class_port).get_node_shared_ptr());
     max_output_boxes = max_output_boxes_input->cast_vector<int64_t>().at(0);
 
     return max_output_boxes;

diff --git a/...e/src/legacy_api/src/transformations/convert_opset1_to_legacy/convert_nms_4_to_legacy.cpp b/...e/src/legacy_api/src/transformations/convert_opset1_to_legacy/convert_nms_4_to_legacy.cpp