From a4b19b42f11b77de49973c337ff77de73e10e206 Mon Sep 17 00:00:00 2001 From: Vladimir Gavrilov Date: Fri, 6 Nov 2020 19:11:04 +0300 Subject: [PATCH] Enable NonMaxSuppression-5 operation and various transformations (#2450) * Some code style fixes. * Started to write the method v5::NonMaxSuppression::evaluate(). * Started to write nGraph reference implementation of NMS-5. * Some additions. * Written preprocessing of boxes data. * Started to write the function non_max_suppression() that calculates NonMaxSuppression-5 operation. * Written postprocessing of the evaluate(). * Small fixes. * Small fix. * Added include for ngraph/shape.hpp. * Written the function intersectionOverUnion. * Some additions. * Small fix. * Continued to write the function non_max_suppression(). * Small fixes. * Small fixes. * Some changes. * Some additions. * Some replacements size_t by int64_t. * Added casts to float in the construction of selected_score variable. * Code style fixes. * Written draft of NMS-5 nGraph reference implementation. * Code style fixes. * Started to write tests for void op::v5::NonMaxSuppression::validate_and_infer_types(). * Added tests for scalars/nonscalars. * Fixes in the test type_prop.nms_v5_output_shape. * Fixes in tests nms_v5_output_shape_2 and nms_v5_output_shape. * Written tests for validate_and_infer_types() of NMS-5. * Code style fixes. * Now NMS-5 evaluate() can have outputs with calculated shapes. * Small fixes. * Corrected tests for NMS-5 validate_and_infer_type(). * Code style fixes. * Started to write inner version of NMS-5 with static output shapes. * Written draft of the inner operation NonMaxSuppressionIE3. * Started to write conversion of op::v5::NonMaxSuppression into NonMaxSuppressionIE3. * Small changes. * Some additions. * Small fixes. * Fixed typo. * Fixed typos. * Written draft of the transformation ConvertNMS5ToLegacyMatcher that converts ngraph::opset5::NonMaxSuppression into op::NonMaxSuppressionIE3. * Written header file for the transformations from NMS-1, NMS-3, NMS-4 to NMS-5. * Started to write conversion of NMS-4 to NMS-5. * Added include for ngraph/opsets/opset4.hpp. * Started to write conversion of NMS-3 to NMS-5. * Small fixes. * Written draft of the conversion of NMS-3 into NMS-5. * Fixed typo. * Started to write conversion of NMS-1 to NMS-5. * Written draft of the conversion NMS-1 to NMS-5. * Started to write tests for the conversion nGraph NMS-5 to inner NMS. * Started to write the test ConvertNMS5ToNMSIEStatic. * Written tests for conversion of nGraph NMS-5 to inner NMSIE3. * Started to write tests for conversion of previous NMS to nGraph NMS-5. * Written tests for conversion of old nGraph NMS to NMS-5. * Started to write tests for opset5::NonMaxSuppression::evaluate(). * Some additions. * Small fix. * Written tests for op::v5::NonMaxSuppression::evaluate(). * Used NGRAPH_RTTI_DECLARATION for NonMaxSuppressionIE3. * Used NGRAPH_RTTI_DECLARATION for NMS-5. * All static local constants and functions for NMS-5 were moved into non-named namespace. * Some code style fixes. * Moved some file. * Small fix. * Code style fix. * Now NMS-5 supports all floating types in inputs 0 and 1. * Moved some files. * Fixed include directive in the file convert_nms_5_to_legacy.cpp with transformations NMS-1, 3, 4 -> NMS-5. * Small changes. * Deleted conversion NMS-3 -> legacy. * Small changes. * Fix in op::v5::NonMaxSuppression::evaluate: output shape [1] instead of [] in the output port 2. * Code style fixes. * Deleted conversion of NMS-4 into legacy NMS. * Deleted redundant ifs. * Added NMS-5 to Python API. * Code style fix. * Small change. * Fixed element type for constants in the conversion of NMS-5 to NMSIE3. * Deleted support of f64 in NMS-5. * Added checks for input element types for inputs #0, #1, #3, #4, #5. * Small change. * Now get_floats throws an exception for unsupported types. * Now nGraph NMS-5 supports 0D and 1D tensors in inputs #2, #3, #4, #5. * Small fix in test_non_max_suppression. * Deleted using namespace std * Fixes in test_non_max_suppression(). * Small fixes. * Added 'import PartialShape' in test_reduction.py. * Deleted creating fake inputs in the ctor of op::v5::NonMaxSuppression. * Deleted creating fake inputs in op::v5::NonMaxSuppression::clone_with_new_inputs. * Corrections in int64_t op::v5::NonMaxSuppression::max_boxes_output_from_input() const. * Corrected functions float op::v5::NonMaxSuppression::soft_nms_sigma_from_input() const, float op::v5::NonMaxSuppression::score_threshold_from_input() const, float op::v5::NonMaxSuppression::iou_threshold_from_input() const. * Small fixes. * Deleted commented code. * Fixes in nms_v5_scalar_inputs_check. * Some changes. * Small fixes. * Code style fixes. * Small changes. * Small changes. * Small fix. * Deleted commented code. * Some refactoring in ConvertNMS4ToNMS5 ctor. * Small fix. * Common part of conversions NMS-1 -> NMS-5, NMS-3 -> NMS-5, NMS-4 -> NMS-5 was moved into the separate function. * Now conversions NMS-1 -> NMS-5, NMS-3 -> NMS-5, NMS-4 -> NMS-5 support both 2 inputs, and 5 inputs. * Now transformations NMS-1 -> NMS-5, NMS-3 -> NMS-5, NMS-4 -> NMS-5 are called from 'umbrella' transformation. * Now the transformation ConvertNMS5ToLegacyMatcher supports NMS-5 with 2, 3, 4, 5, or 6 inputs. * The transformation ConvertNMS5ToLegacyMatcher was rewritten using Reshape instead of Unsqueeze. * Started to rewrite tests for the transformation ConvertNMS5ToLegacyMatcher. * Some fixes. * Small fixes. * Corrected tests for the transformation NMS-5 -> NMSIE3. * Small formatting fix. * Now methods max_boxes_output_from_input(), iou_threshold_from_input(), score_threshold_from_input(), soft_nms_sigma_from_input() of op::v5::NunMaxSuppression are public. * Started to move op::v5::NonMaxSuppression::evaluate() into ngraph/test/runtime/interpreter. * Added NMS-5 to ngraph/test/runtime/interpreter/int_executable. * Small fixes. * Code style fixes. * Written draft test nonmaxsuppression_center_point_box_format_backend in ngraph/test/backend. * Small fix. * Written draft tests of NonMaxSuppression in ngraph/test/runtime. * Some changes. * Small changes. * Disabled IE_CPU tests for NMS-5. * Deleted op_eval tests for NMS-5. * Deleted evaluate() method of NMS-5. * Now all nGraph functions in tests of the transformation NMS-5 -> NMSIE3 have one output. * Now preprocessing and postprocessing of the calculation of NMS-5 in reference implementation. * Code style fixes. * Some fixes in tests for the transformation NMS-5 -> NMSIE3. * Replaced precision i64 -> i32 for some constants in tests for the transformation NMS-5 -> NMSIE3. * Written creating CNNLayer for NMS-5. * Added creating CNNLayer for NonMaxSuppressionIE3. * some changes. * Now conversions NMS-1, NMS-3, NMS-4 -> NMS-5 and NMS-5 -> NMSIE3 generate NMS nodes with 5 inputs. * Fixed ctor in MKLDNN NonMaxSuppressionImpl: validation of number of output edges. * Added conversion of output_type for NMS-5. * Fixes in the transformation NMS5 -> NMSIE3. * Fixes in the conversion of NMS-5 to NMSIE3. * Fixes in MKLDNN NMS ctor. * Small fix. * Fixed tests for the transformation NMS5 -> NMSIE3. * Fixed tests for conversions NMS-1, NMS-3, NMS-4 -> NMS-5. * Small fixes in MKLDNN NMS ctor. * Rewritten create_layer() functions for NMS-5 and NMSIE3 as addSpecificCreator() functions. * Disabled tests for IE IR reader for NMS-4. * Deleted debug code. * Added comment about disabling tests IE_CPU.nonmaxsuppression. * Written IE IR Reader test for NMS-4. * Deleted function float_from_constant_node. * Small fixes. * Deleted functions function_from_model and construct_weights. * Small fix. * Replaced push_back with emplace_back in the conversion of NMS-5 to legacy. * Small changes. * Some fixes. * Refactored reference implementation of NMS-5. * Moved structure NMSAttributes to unnamed namespace. * Code style fixes. * Small fix. --- .../src/legacy_api/include/legacy/ie_layers.h | 4 + .../include/legacy/ngraph_ops/nms_ie.hpp | 9 + .../convert_nms_4_to_legacy.hpp | 33 - .../convert_nms_5_to_legacy.hpp} | 18 +- .../src/convert_function_to_cnn_network.cpp | 77 ++ .../legacy_api/src/ie_layer_validators.cpp | 1 + .../src/legacy_api/src/ngraph_ops/nms_ie.cpp | 22 +- .../convert_nms_4_to_legacy.cpp | 113 --- .../convert_nms_5_to_legacy.cpp | 110 +++ .../convert_opset1_to_legacy.cpp | 4 +- .../ir_reader_v7/ie_layer_validators.cpp | 4 +- .../convert_previous_nms_to_nms_5.hpp | 50 ++ .../common_optimizations.cpp | 3 + .../src/transformations/convert_precision.cpp | 11 + .../op_conversions/convert_nms3.cpp | 45 -- .../convert_previous_nms_to_nms_5.cpp | 209 +++++ .../convert_opset3_to_opset2.cpp | 2 - .../non_max_suppression_tests.cpp | 407 ++++++++-- .../transformations/convert_nms3_test.cpp | 62 -- .../transformations/convert_nms4_test.cpp | 150 ---- .../transformations/convert_nms5_test.cpp | 763 ++++++++++++++++++ .../convert_previous_nms_to_nms_5.cpp | 228 ++++++ .../include/ngraph/op/non_max_suppression.hpp | 11 +- .../runtime/reference/non_max_suppression.hpp | 81 ++ .../runtime/reference/non_max_suppression.cpp | 574 +++++++++++++ ngraph/core/src/op/non_max_suppression.cpp | 207 +++-- ngraph/python/src/ngraph/opset5/__init__.py | 2 +- ngraph/python/src/ngraph/opset5/ops.py | 52 ++ .../tests/test_ngraph/test_reduction.py | 8 +- ngraph/test/CMakeLists.txt | 1 + .../test/backend/non_max_suppression.in.cpp | 614 ++++++++++++++ ngraph/test/runtime/ie/unit_test.manifest | 20 + .../runtime/interpreter/int_executable.cpp | 3 + .../runtime/interpreter/int_executable.hpp | 38 + .../runtime/interpreter/opset_int_tbl.hpp | 1 + ngraph/test/type_prop/non_max_suppression.cpp | 23 +- 36 files changed, 3399 insertions(+), 561 deletions(-) delete mode 100644 inference-engine/src/legacy_api/include/legacy/transformations/convert_opset1_to_legacy/convert_nms_4_to_legacy.hpp rename inference-engine/src/{transformations/include/transformations/op_conversions/convert_nms3.hpp => legacy_api/include/legacy/transformations/convert_opset1_to_legacy/convert_nms_5_to_legacy.hpp} (58%) delete mode 100644 inference-engine/src/legacy_api/src/transformations/convert_opset1_to_legacy/convert_nms_4_to_legacy.cpp create mode 100644 inference-engine/src/legacy_api/src/transformations/convert_opset1_to_legacy/convert_nms_5_to_legacy.cpp create mode 100644 inference-engine/src/transformations/include/transformations/op_conversions/convert_previous_nms_to_nms_5.hpp delete mode 100644 inference-engine/src/transformations/src/transformations/op_conversions/convert_nms3.cpp create mode 100644 inference-engine/src/transformations/src/transformations/op_conversions/convert_previous_nms_to_nms_5.cpp delete mode 100644 inference-engine/tests/functional/inference_engine/transformations/convert_nms3_test.cpp delete mode 100644 inference-engine/tests/functional/inference_engine/transformations/convert_nms4_test.cpp create mode 100644 inference-engine/tests/functional/inference_engine/transformations/convert_nms5_test.cpp create mode 100644 inference-engine/tests/functional/inference_engine/transformations/convert_previous_nms_to_nms_5.cpp create mode 100644 ngraph/core/reference/include/ngraph/runtime/reference/non_max_suppression.hpp create mode 100644 ngraph/core/reference/src/runtime/reference/non_max_suppression.cpp create mode 100644 ngraph/test/backend/non_max_suppression.in.cpp diff --git a/inference-engine/src/legacy_api/include/legacy/ie_layers.h b/inference-engine/src/legacy_api/include/legacy/ie_layers.h index d02532dbcc7bd4..b298ac969fa6b3 100644 --- a/inference-engine/src/legacy_api/include/legacy/ie_layers.h +++ b/inference-engine/src/legacy_api/include/legacy/ie_layers.h @@ -2064,6 +2064,10 @@ class INFERENCE_ENGINE_INTERNAL_CNNLAYER_CLASS(NonMaxSuppressionLayer): public C * classes */ bool sort_result_descending = true; + /** + * @brief Output type for first and third inputs + */ + std::string output_type = "I64"; /** * @brief Creates a new NonMaxSuppressionLayer instance. */ diff --git a/inference-engine/src/legacy_api/include/legacy/ngraph_ops/nms_ie.hpp b/inference-engine/src/legacy_api/include/legacy/ngraph_ops/nms_ie.hpp index 49e689e9bd4727..f8b04c5fcca48d 100644 --- a/inference-engine/src/legacy_api/include/legacy/ngraph_ops/nms_ie.hpp +++ b/inference-engine/src/legacy_api/include/legacy/ngraph_ops/nms_ie.hpp @@ -62,6 +62,15 @@ class INFERENCE_ENGINE_API_CLASS(NonMaxSuppressionIE3) : public Op { public: NGRAPH_RTTI_DECLARATION; + NonMaxSuppressionIE3(const Output& boxes, + const Output& scores, + const Output& max_output_boxes_per_class, + const Output& iou_threshold, + const Output& score_threshold, + int center_point_box, + bool sort_result_descending, + const ngraph::element::Type& output_type = ngraph::element::i64); + NonMaxSuppressionIE3(const Output& boxes, const Output& scores, const Output& max_output_boxes_per_class, diff --git a/inference-engine/src/legacy_api/include/legacy/transformations/convert_opset1_to_legacy/convert_nms_4_to_legacy.hpp b/inference-engine/src/legacy_api/include/legacy/transformations/convert_opset1_to_legacy/convert_nms_4_to_legacy.hpp deleted file mode 100644 index affd092ce56b0a..00000000000000 --- a/inference-engine/src/legacy_api/include/legacy/transformations/convert_opset1_to_legacy/convert_nms_4_to_legacy.hpp +++ /dev/null @@ -1,33 +0,0 @@ -// Copyright (C) 2018-2020 Intel Corporation -// SPDX-License-Identifier: Apache-2.0 -// - -#pragma once - -#include -#include - -#include - -#include - -namespace ngraph { -namespace pass { - - class INFERENCE_ENGINE_API_CLASS(ConvertNMS4ToLegacyMatcher); - -} // namespace pass -} // namespace ngraph - -/* - * Description: - * Convert NMS-4 directly to legacy NMS because NMS-3 and NMS-1 have different shape infer function - */ - - -class ngraph::pass::ConvertNMS4ToLegacyMatcher: public ngraph::pass::MatcherPass { -public: - NGRAPH_RTTI_DECLARATION; - ConvertNMS4ToLegacyMatcher(); -}; - diff --git a/inference-engine/src/transformations/include/transformations/op_conversions/convert_nms3.hpp b/inference-engine/src/legacy_api/include/legacy/transformations/convert_opset1_to_legacy/convert_nms_5_to_legacy.hpp similarity index 58% rename from inference-engine/src/transformations/include/transformations/op_conversions/convert_nms3.hpp rename to inference-engine/src/legacy_api/include/legacy/transformations/convert_opset1_to_legacy/convert_nms_5_to_legacy.hpp index 724566fdd2391d..375ec6cdc71ada 100644 --- a/inference-engine/src/transformations/include/transformations/op_conversions/convert_nms3.hpp +++ b/inference-engine/src/legacy_api/include/legacy/transformations/convert_opset1_to_legacy/convert_nms_5_to_legacy.hpp @@ -14,18 +14,20 @@ namespace ngraph { namespace pass { -class TRANSFORMATIONS_API ConvertNMS1ToNMS3; +class INFERENCE_ENGINE_API_CLASS(ConvertNMS5ToLegacyMatcher); } // namespace pass } // namespace ngraph -class ngraph::pass::ConvertNMS1ToNMS3: public ngraph::pass::GraphRewrite { +/* + * Description: + * Convert NMS-5 directly to inner NMS. + */ + + +class ngraph::pass::ConvertNMS5ToLegacyMatcher: public ngraph::pass::MatcherPass { public: NGRAPH_RTTI_DECLARATION; - ConvertNMS1ToNMS3() : GraphRewrite() { - convert_nms1_to_nms3(); - } - -private: - void convert_nms1_to_nms3(); + ConvertNMS5ToLegacyMatcher(); }; + diff --git a/inference-engine/src/legacy_api/src/convert_function_to_cnn_network.cpp b/inference-engine/src/legacy_api/src/convert_function_to_cnn_network.cpp index a90a9f3a73611e..3e8a8130ed5f48 100644 --- a/inference-engine/src/legacy_api/src/convert_function_to_cnn_network.cpp +++ b/inference-engine/src/legacy_api/src/convert_function_to_cnn_network.cpp @@ -653,6 +653,83 @@ InferenceEngine::details::CNNLayerCreator::CNNLayerCreator(const std::shared_ptr }); + addSpecificCreator({"NonMaxSuppressionIE3"}, [](const std::shared_ptr<::ngraph::Node>& node, + const std::map& params) -> CNNLayerPtr { + LayerParams attrs = {node->get_friendly_name(), "NonMaxSuppression", + details::convertPrecision(node->get_output_element_type(0))}; + + auto castedLayer = ::ngraph::as_type_ptr<::ngraph::op::NonMaxSuppressionIE3>(node); + IE_ASSERT(castedLayer) << " Operation " << node->description() << " with name " + << node->get_friendly_name() << " cannot be casted to ngraph::op::NonMaxSuppressionIE3"; + + auto res = std::make_shared(attrs); + res->params = params; + + res->params["center_point_box"] = castedLayer->m_center_point_box ? "true" : "false"; + res->params["sort_result_descending"] = castedLayer->m_sort_result_descending ? "true" : "false"; + + auto output_type = details::convertPrecision(castedLayer->m_output_type); + std::string output_type_str; + switch (output_type) { + case Precision::I32: + output_type_str = "I32"; + break; + case Precision::I64: + output_type_str = "I64"; + break; + default: + THROW_IE_EXCEPTION << "Unsupported output type"; + } + res->params["output_type"] = output_type_str; + + return res; + }); + + addSpecificCreator({"NonMaxSuppression"}, [](const std::shared_ptr<::ngraph::Node>& node, + const std::map& params) -> CNNLayerPtr { + LayerParams attrs = {node->get_friendly_name(), "NonMaxSuppression", + details::convertPrecision(node->get_output_element_type(0))}; + + auto castedLayer = ::ngraph::as_type_ptr<::ngraph::op::v5::NonMaxSuppression>(node); + IE_ASSERT(castedLayer) << " Operation " << node->description() << " with name " + << node->get_friendly_name() << " cannot be casted to ngraph::op::v5::NonMaxSuppression"; + + auto res = std::make_shared(attrs); + res->params = params; + + auto box_encoding = castedLayer->get_box_encoding(); + switch (box_encoding) { + case ngraph::op::v5::NonMaxSuppression::BoxEncodingType::CORNER: + res->params["center_point_box"] = "false"; + break; + case ngraph::op::v5::NonMaxSuppression::BoxEncodingType::CENTER: + res->params["center_point_box"] = "true"; + break; + default: + THROW_IE_EXCEPTION << "Unsupported box encoding for NonMaxSuppression op"; + break; + } + + auto output_type = details::convertPrecision(castedLayer->get_output_type()); + std::string output_type_str; + switch (output_type) { + case Precision::I32: + output_type_str = "I32"; + break; + case Precision::I64: + output_type_str = "I64"; + break; + default: + THROW_IE_EXCEPTION << "Unsupported output type"; + } + res->params["output_type"] = output_type_str; + + bool sort_result_descending = castedLayer->get_sort_result_descending(); + res->params["sort_result_descending"] = sort_result_descending ? "true" : "false"; + + return res; + }); + addSpecificCreator({"NonMaxSuppressionIE"}, [](const std::shared_ptr<::ngraph::Node>& node, const std::map& params) -> CNNLayerPtr { LayerParams attrs = {node->get_friendly_name(), "NonMaxSuppression", details::convertPrecision(node->get_output_element_type(0))}; diff --git a/inference-engine/src/legacy_api/src/ie_layer_validators.cpp b/inference-engine/src/legacy_api/src/ie_layer_validators.cpp index ee9757a9e9e608..86b7396e4998cd 100644 --- a/inference-engine/src/legacy_api/src/ie_layer_validators.cpp +++ b/inference-engine/src/legacy_api/src/ie_layer_validators.cpp @@ -1181,6 +1181,7 @@ void NMSValidator::parseParams(CNNLayer* layer) { casted->center_point_box = layer->GetParamAsBool("center_point_box", false); casted->sort_result_descending = layer->GetParamAsBool("sort_result_descending", true); + casted->output_type = layer->GetParamAsString("output_type", "I64"); } #define REG_LAYER_VALIDATOR_FOR_TYPE(__validator, __type) _validators[#__type] = std::make_shared<__validator>(#__type) diff --git a/inference-engine/src/legacy_api/src/ngraph_ops/nms_ie.cpp b/inference-engine/src/legacy_api/src/ngraph_ops/nms_ie.cpp index 4e838696811e9b..be26e1c291e151 100644 --- a/inference-engine/src/legacy_api/src/ngraph_ops/nms_ie.cpp +++ b/inference-engine/src/legacy_api/src/ngraph_ops/nms_ie.cpp @@ -102,7 +102,20 @@ void op::NonMaxSuppressionIE2::validate_and_infer_types() { set_output_type(0, nms->output(0).get_element_type(), nms->output(0).get_partial_shape()); } -NGRAPH_RTTI_DEFINITION(op::NonMaxSuppressionIE3, "NonMaxSuppressionIE", 3); +NGRAPH_RTTI_DEFINITION(op::NonMaxSuppressionIE3, "NonMaxSuppressionIE3", 3); + +op::NonMaxSuppressionIE3::NonMaxSuppressionIE3(const Output& boxes, + const Output& scores, + const Output& max_output_boxes_per_class, + const Output& iou_threshold, + const Output& score_threshold, + int center_point_box, + bool sort_result_descending, + const ngraph::element::Type& output_type) + : Op({boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold}), + m_center_point_box(center_point_box), m_sort_result_descending(sort_result_descending), m_output_type(output_type) { + constructor_validate_and_infer_types(); +} op::NonMaxSuppressionIE3::NonMaxSuppressionIE3(const Output& boxes, const Output& scores, @@ -139,8 +152,13 @@ static constexpr size_t max_output_boxes_per_class_port = 2; int64_t op::NonMaxSuppressionIE3::max_boxes_output_from_input() const { int64_t max_output_boxes{0}; + size_t num_of_inputs = inputs().size(); + if (num_of_inputs < 3) { + return 0; + } + const auto max_output_boxes_input = - as_type_ptr(input_value(2).get_node_shared_ptr()); + as_type_ptr(input_value(max_output_boxes_per_class_port).get_node_shared_ptr()); max_output_boxes = max_output_boxes_input->cast_vector().at(0); return max_output_boxes; diff --git a/inference-engine/src/legacy_api/src/transformations/convert_opset1_to_legacy/convert_nms_4_to_legacy.cpp b/inference-engine/src/legacy_api/src/transformations/convert_opset1_to_legacy/convert_nms_4_to_legacy.cpp deleted file mode 100644 index ca6a7459b63952..00000000000000 --- a/inference-engine/src/legacy_api/src/transformations/convert_opset1_to_legacy/convert_nms_4_to_legacy.cpp +++ /dev/null @@ -1,113 +0,0 @@ -// Copyright (C) 2018-2020 Intel Corporation -// SPDX-License-Identifier: Apache-2.0 -// - -#include -#include - -#include -#include -#include -#include -#include -#include - -#include "legacy/transformations/convert_opset1_to_legacy/convert_nms_4_to_legacy.hpp" - -NGRAPH_RTTI_DEFINITION(ngraph::pass::ConvertNMS4ToLegacyMatcher, "ConvertNMS4ToLegacyMatcher", 0); - -ngraph::pass::ConvertNMS4ToLegacyMatcher::ConvertNMS4ToLegacyMatcher() { - auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); - auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); - auto max_output_boxes_per_class = ngraph::opset4::Constant::create(element::i64, Shape{}, {10}); - auto iou_threshold = ngraph::opset4::Constant::create(element::f32, Shape{}, {0.75}); - auto score_threshold = ngraph::opset4::Constant::create(element::f32, Shape{}, {0.7}); - auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, - iou_threshold, score_threshold); - - ngraph::matcher_pass_callback callback = [](pattern::Matcher &m) { - auto nms_4 = std::dynamic_pointer_cast(m.get_match_root()); - if (!nms_4) { - return false; - } - - const auto new_args = nms_4->input_values(); - const auto& arg2 = new_args.size() > 2 ? new_args.at(2) : ngraph::opset4::Constant::create(element::i32, Shape{}, {0}); - const auto& arg3 = new_args.size() > 3 ? new_args.at(3) : ngraph::opset4::Constant::create(element::f32, Shape{}, {.0f}); - const auto& arg4 = new_args.size() > 4 ? new_args.at(4) : ngraph::opset4::Constant::create(element::f32, Shape{}, {.0f}); - - const auto max_output_boxes_per_class_rank = arg2.get_partial_shape().rank(); - const auto iou_threshold_rank = arg3.get_partial_shape().rank(); - const auto score_threshold_rank = arg4.get_partial_shape().rank(); - - // Check that required ranks are not dynamic - if (max_output_boxes_per_class_rank.is_dynamic() || - iou_threshold_rank.is_dynamic() || - score_threshold_rank.is_dynamic()) { - return false; - } - - if (max_output_boxes_per_class_rank.get_length() == 1 && - iou_threshold_rank.get_length() == 1 && - score_threshold_rank.get_length() == 1) { - return false; - } - - // vector of new nGraph operations - NodeVector new_ops; - - auto new_max_per_class = arg2; - if (max_output_boxes_per_class_rank.get_length() == 0) { - // WA: we need to create Constant manually because it requires by NMS shape inference - // otherwise we will get dynamic shape until first CF is executed. It can be resolved - // if CF will be executed right after transformation and before Validate pass. - if (auto new_max_per_class_const = std::dynamic_pointer_cast(new_max_per_class.get_node_shared_ptr())) { - new_max_per_class = opset1::Constant::create(element::i64, Shape{1}, new_max_per_class_const->cast_vector()); - } else { - new_max_per_class = std::make_shared(arg2, opset1::Constant::create(element::i64, Shape{1}, {0})); - new_ops.push_back(new_max_per_class.get_node_shared_ptr()); - } - } - auto new_iou_threshold = arg3; - if (iou_threshold_rank.get_length() == 0) { - new_iou_threshold = std::make_shared(arg3, opset1::Constant::create(element::i64, Shape{1}, {0})); - new_ops.push_back(new_iou_threshold.get_node_shared_ptr()); - } - auto new_score_threshold = arg4; - if (score_threshold_rank.get_length() == 0) { - new_score_threshold = std::make_shared(arg4, opset1::Constant::create(element::i64, Shape{1}, {0})); - new_ops.push_back(new_score_threshold.get_node_shared_ptr()); - } - - int center_point_box = 0; - switch (nms_4->get_box_encoding()) { - case ::ngraph::opset4::NonMaxSuppression::BoxEncodingType::CENTER: - center_point_box = 1; - break; - case ::ngraph::opset4::NonMaxSuppression::BoxEncodingType::CORNER: - center_point_box = 0; - break; - default: - throw ngraph_error("NonMaxSuppression layer " + nms_4->get_friendly_name() + - " has unsupported box encoding"); - } - const auto nms_legacy = std::make_shared( - new_args.at(0), - new_args.at(1), - new_max_per_class, - new_iou_threshold, - new_score_threshold, - center_point_box, - nms_4->get_sort_result_descending(), - nms_4->get_output_type()); - new_ops.push_back(nms_legacy); - - nms_legacy->set_friendly_name(nms_4->get_friendly_name()); - ngraph::copy_runtime_info(nms_4, new_ops); - ngraph::replace_node(nms_4, nms_legacy); - return true; - }; - - auto m = std::make_shared(nms, "ConvertNMS4ToNMSLegacy"); - this->register_matcher(m, callback); -} diff --git a/inference-engine/src/legacy_api/src/transformations/convert_opset1_to_legacy/convert_nms_5_to_legacy.cpp b/inference-engine/src/legacy_api/src/transformations/convert_opset1_to_legacy/convert_nms_5_to_legacy.cpp new file mode 100644 index 00000000000000..8ef630e16879d0 --- /dev/null +++ b/inference-engine/src/legacy_api/src/transformations/convert_opset1_to_legacy/convert_nms_5_to_legacy.cpp @@ -0,0 +1,110 @@ +// Copyright (C) 2018-2020 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include +#include + +#include +#include +#include +#include +#include +#include +#include + +#include "legacy/transformations/convert_opset1_to_legacy/convert_nms_5_to_legacy.hpp" + +NGRAPH_RTTI_DEFINITION(ngraph::pass::ConvertNMS5ToLegacyMatcher, "ConvertNMS5ToLegacyMatcher", 0); + +ngraph::pass::ConvertNMS5ToLegacyMatcher::ConvertNMS5ToLegacyMatcher() { + auto nms = ngraph::pattern::wrap_type(); + + ngraph::matcher_pass_callback callback = [](pattern::Matcher &m) { + auto nms_5 = std::dynamic_pointer_cast(m.get_match_root()); + if (!nms_5) { + return false; + } + + const auto new_args = nms_5->input_values(); + const std::size_t num_of_inputs = new_args.size(); + + const auto& arg2 = num_of_inputs > 2 ? new_args.at(2) : ngraph::opset5::Constant::create(element::i32, Shape{}, {0}); + const auto& arg3 = num_of_inputs > 3 ? new_args.at(3) : ngraph::opset5::Constant::create(element::f32, Shape{}, {.0f}); + const auto& arg4 = num_of_inputs > 4 ? new_args.at(4) : ngraph::opset5::Constant::create(element::f32, Shape{}, {.0f}); + + // vector of new nGraph operations + NodeVector new_ops; + + auto one_dim_shape = Shape{1}; + + Output new_max_per_class; + Output new_iou_threshold; + Output new_score_threshold; + Output new_soft_nms_sigma; + + Output new_shape_for_max_per_class = opset1::Constant::create(ngraph::element::i64, Shape{1}, {1}); + Output new_shape_for_iou_threshold = opset1::Constant::create(ngraph::element::i64, Shape{1}, {1}); + Output new_shape_for_score_threshold = opset1::Constant::create(ngraph::element::i64, Shape{1}, {1}); + Output new_shape_for_soft_nms_sigma = opset1::Constant::create(ngraph::element::i64, Shape{1}, {1}); + + new_max_per_class = std::make_shared(arg2, new_shape_for_max_per_class, true); + new_ops.emplace_back(new_max_per_class.get_node_shared_ptr()); + + new_iou_threshold = std::make_shared(arg3, new_shape_for_iou_threshold, true); + new_ops.emplace_back(new_iou_threshold.get_node_shared_ptr()); + + new_score_threshold = std::make_shared(arg4, new_shape_for_score_threshold, true); + new_ops.emplace_back(new_score_threshold.get_node_shared_ptr()); + + int center_point_box = 0; + switch (nms_5->get_box_encoding()) { + case ::ngraph::opset5::NonMaxSuppression::BoxEncodingType::CENTER: + center_point_box = 1; + break; + case ::ngraph::opset5::NonMaxSuppression::BoxEncodingType::CORNER: + center_point_box = 0; + break; + default: + throw ngraph_error("NonMaxSuppression layer " + nms_5->get_friendly_name() + + " has unsupported box encoding"); + } + + std::shared_ptr nms_legacy{nullptr}; + + if (num_of_inputs > 5 && nms_5->soft_nms_sigma_from_input() != 0.0f) { + new_soft_nms_sigma = std::make_shared(new_args.at(5), new_shape_for_soft_nms_sigma, true); + new_ops.emplace_back(new_soft_nms_sigma.get_node_shared_ptr()); + nms_legacy = std::make_shared( + new_args.at(0), + new_args.at(1), + new_max_per_class, + new_iou_threshold, + new_score_threshold, + new_soft_nms_sigma, + center_point_box, + nms_5->get_sort_result_descending(), + nms_5->get_output_type()); + new_ops.emplace_back(nms_legacy); + } else { + nms_legacy = std::make_shared( + new_args.at(0), + new_args.at(1), + new_max_per_class, + new_iou_threshold, + new_score_threshold, + center_point_box, + nms_5->get_sort_result_descending(), + nms_5->get_output_type()); + new_ops.emplace_back(nms_legacy); + } + + nms_legacy->set_friendly_name(nms_5->get_friendly_name()); + ngraph::copy_runtime_info(nms_5, new_ops); + ngraph::replace_node(nms_5, nms_legacy); + return true; + }; + + auto m = std::make_shared(nms, "ConvertNMS5ToNMSLegacy"); + this->register_matcher(m, callback); +} diff --git a/inference-engine/src/legacy_api/src/transformations/convert_opset1_to_legacy/convert_opset1_to_legacy.cpp b/inference-engine/src/legacy_api/src/transformations/convert_opset1_to_legacy/convert_opset1_to_legacy.cpp index 53ce548e8bf6cc..4182ac11d92716 100644 --- a/inference-engine/src/legacy_api/src/transformations/convert_opset1_to_legacy/convert_opset1_to_legacy.cpp +++ b/inference-engine/src/legacy_api/src/transformations/convert_opset1_to_legacy/convert_opset1_to_legacy.cpp @@ -13,7 +13,7 @@ #include "legacy/transformations/convert_opset1_to_legacy/convert_mul_add_to_scaleshift_or_power.hpp" #include "legacy/transformations/convert_opset1_to_legacy/convert_mul_or_add_finally.hpp" #include "legacy/transformations/convert_opset1_to_legacy/convert_nms_to_nms_ie.hpp" -#include "legacy/transformations/convert_opset1_to_legacy/convert_nms_4_to_legacy.hpp" +#include "legacy/transformations/convert_opset1_to_legacy/convert_nms_5_to_legacy.hpp" #include "legacy/transformations/convert_opset1_to_legacy/convert_normalizel2_to_normalize_ie.hpp" #include "legacy/transformations/convert_opset1_to_legacy/convert_one_hot_to_one_hot_ie.hpp" #include "legacy/transformations/convert_opset1_to_legacy/convert_pad_to_pad_ie.hpp" @@ -133,7 +133,7 @@ bool ngraph::pass::ConvertOpSet1ToLegacy::run_on_function(std::shared_ptradd_matcher(); anchor->add_matcher(); anchor->add_matcher(); - anchor->add_matcher(); + anchor->add_matcher(); anchor->add_matcher(); anchor->add_matcher(); anchor->add_matcher(); diff --git a/inference-engine/src/readers/ir_reader_v7/ie_layer_validators.cpp b/inference-engine/src/readers/ir_reader_v7/ie_layer_validators.cpp index 3d3b89a4264499..245e87a9042362 100644 --- a/inference-engine/src/readers/ir_reader_v7/ie_layer_validators.cpp +++ b/inference-engine/src/readers/ir_reader_v7/ie_layer_validators.cpp @@ -2098,9 +2098,9 @@ void NMSValidator::checkParams(const CNNLayer* layer) { void NMSValidator::checkShapes(const CNNLayer* layer, const vector& inShapes) const { size_t numInputs = inShapes.size(); - if (numInputs < 2 || numInputs > 5) + if (numInputs < 2 || numInputs > 6) THROW_IE_EXCEPTION << layer->name - << " NonMaxSuppression can take 2 - 5 inputs, but actually it has: " << numInputs; + << " NonMaxSuppression can take 2 - 6 inputs, but actually it has: " << numInputs; if (inShapes[0].size() != 3 || inShapes[0][2] != 4) THROW_IE_EXCEPTION << layer->name << " 'boxes' should be with shape [num_batches, spatial_dimension, 4]"; diff --git a/inference-engine/src/transformations/include/transformations/op_conversions/convert_previous_nms_to_nms_5.hpp b/inference-engine/src/transformations/include/transformations/op_conversions/convert_previous_nms_to_nms_5.hpp new file mode 100644 index 00000000000000..f0296717526cae --- /dev/null +++ b/inference-engine/src/transformations/include/transformations/op_conversions/convert_previous_nms_to_nms_5.hpp @@ -0,0 +1,50 @@ +// Copyright (C) 2018-2020 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include +#include + +#include +#include + +namespace ngraph { +namespace pass { + +class TRANSFORMATIONS_API ConvertNMS1ToNMS5; +class TRANSFORMATIONS_API ConvertNMS3ToNMS5; +class TRANSFORMATIONS_API ConvertNMS4ToNMS5; +class TRANSFORMATIONS_API ConvertPreviousNMSToNMS5; + +} // namespace pass +} // namespace ngraph + +class ngraph::pass::ConvertNMS1ToNMS5: public ngraph::pass::MatcherPass { +public: + NGRAPH_RTTI_DECLARATION; + ConvertNMS1ToNMS5(); +}; + +class ngraph::pass::ConvertNMS3ToNMS5: public ngraph::pass::MatcherPass { +public: + NGRAPH_RTTI_DECLARATION; + ConvertNMS3ToNMS5(); +}; + +class ngraph::pass::ConvertNMS4ToNMS5: public ngraph::pass::MatcherPass { +public: + NGRAPH_RTTI_DECLARATION; + ConvertNMS4ToNMS5(); +}; + +class ngraph::pass::ConvertPreviousNMSToNMS5: public ngraph::pass::GraphRewrite { +public: + NGRAPH_RTTI_DECLARATION; + ConvertPreviousNMSToNMS5() { + add_matcher(); + add_matcher(); + add_matcher(); + } +}; diff --git a/inference-engine/src/transformations/src/transformations/common_optimizations/common_optimizations.cpp b/inference-engine/src/transformations/src/transformations/common_optimizations/common_optimizations.cpp index 4f3a264c262b57..3da68fccdb5639 100644 --- a/inference-engine/src/transformations/src/transformations/common_optimizations/common_optimizations.cpp +++ b/inference-engine/src/transformations/src/transformations/common_optimizations/common_optimizations.cpp @@ -42,6 +42,7 @@ #include "transformations/op_conversions/reduce_l1_decomposition.hpp" #include "transformations/op_conversions/reduce_l2_decomposition.hpp" #include "transformations/op_conversions/hswish_decomposition.hpp" +#include "transformations/op_conversions/convert_previous_nms_to_nms_5.hpp" #include "transformations/op_conversions/hsigmoid_decomposition.hpp" #include "transformations/op_conversions/log_softmax_decomposition.hpp" @@ -112,6 +113,8 @@ bool ngraph::pass::CommonOptimizations::run_on_function(std::shared_ptr(); manager.register_pass(); + manager.register_pass(); + auto fq_fusions = manager.register_pass(); fq_fusions->add_matcher(); fq_fusions->add_matcher(); diff --git a/inference-engine/src/transformations/src/transformations/convert_precision.cpp b/inference-engine/src/transformations/src/transformations/convert_precision.cpp index bb7aeaca89fc15..c08fe61dbd841f 100644 --- a/inference-engine/src/transformations/src/transformations/convert_precision.cpp +++ b/inference-engine/src/transformations/src/transformations/convert_precision.cpp @@ -7,6 +7,7 @@ #include #include +#include #include #include #include @@ -21,6 +22,7 @@ bool fuse_type_to_parameter(std::shared_ptr & node, ngraph::elemen bool fuse_type_to_convert(std::shared_ptr & node, ngraph::element::Type to, size_t idx); bool fuse_type_to_nms3(std::shared_ptr & node, ngraph::element::Type to, size_t idx); bool fuse_type_to_nms4(std::shared_ptr & node, ngraph::element::Type to, size_t idx); +bool fuse_type_to_nms5(std::shared_ptr & node, ngraph::element::Type to, size_t idx); bool fuse_type_to_topk(std::shared_ptr & node, ngraph::element::Type to, size_t idx); bool fuse_type_to_nonzero(std::shared_ptr & node, ngraph::element::Type to, size_t idx); bool fuse_type_to_bucketize(std::shared_ptr & node, ngraph::element::Type to, size_t idx); @@ -81,6 +83,7 @@ bool ngraph::pass::ConvertPrecision::run_on_function(std::shared_ptr & node, ngraph::element::Ty return false; } +bool fuse_type_to_nms5(std::shared_ptr & node, ngraph::element::Type to, size_t idx) { + if (auto nms = as_type_ptr(node)) { + nms->set_output_type(to); + return true; + } + return false; +} + bool fuse_type_to_topk(std::shared_ptr & node, ngraph::element::Type to, size_t idx) { if (auto topk = as_type_ptr(node)) { if (idx == 1 && (to == element::i32 || to == element::i64)) { diff --git a/inference-engine/src/transformations/src/transformations/op_conversions/convert_nms3.cpp b/inference-engine/src/transformations/src/transformations/op_conversions/convert_nms3.cpp deleted file mode 100644 index fa90efa9c58c81..00000000000000 --- a/inference-engine/src/transformations/src/transformations/op_conversions/convert_nms3.cpp +++ /dev/null @@ -1,45 +0,0 @@ -// Copyright (C) 2018-2020 Intel Corporation -// SPDX-License-Identifier: Apache-2.0 -// - -#include "transformations/op_conversions/convert_nms3.hpp" - -#include -#include - -#include -#include -#include -#include - -NGRAPH_RTTI_DEFINITION(ngraph::pass::ConvertNMS1ToNMS3, "ConvertNMS1ToNMS3", 0); - -void ngraph::pass::ConvertNMS1ToNMS3::convert_nms1_to_nms3() { - auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); - auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); - auto max_output_boxes_per_class = ngraph::opset3::Constant::create(element::i64, Shape{}, {10}); - auto iou_threshold = ngraph::opset3::Constant::create(element::f32, Shape{}, {0.75}); - auto score_threshold = ngraph::opset3::Constant::create(element::f32, Shape{}, {0.7}); - auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, - iou_threshold, score_threshold); - - ngraph::graph_rewrite_callback callback = [](pattern::Matcher &m) { - auto nms = std::dynamic_pointer_cast(m.get_match_root()); - if (!nms) { - return false; - } - - auto nms3 = std::make_shared(nms->input_value(0), nms->input_value(1), - nms->input_value(2), nms->input_value(3), nms->input_value(4), - static_cast(nms->get_box_encoding()), - nms->get_sort_result_descending()); - - nms3->set_friendly_name(nms->get_friendly_name()); - ngraph::copy_runtime_info(nms, nms3); - ngraph::replace_node(nms, nms3); - return true; - }; - - auto m = std::make_shared(nms, "ConvertNMS1ToNMS3"); - this->add_matcher(m, callback, PassProperty::CHANGE_DYNAMIC_STATE); -} diff --git a/inference-engine/src/transformations/src/transformations/op_conversions/convert_previous_nms_to_nms_5.cpp b/inference-engine/src/transformations/src/transformations/op_conversions/convert_previous_nms_to_nms_5.cpp new file mode 100644 index 00000000000000..3328f02058d2a7 --- /dev/null +++ b/inference-engine/src/transformations/src/transformations/op_conversions/convert_previous_nms_to_nms_5.cpp @@ -0,0 +1,209 @@ +// Copyright (C) 2018-2020 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include "transformations/op_conversions/convert_previous_nms_to_nms_5.hpp" + +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +using namespace ngraph; + +namespace { +struct NMSAttributes { + ngraph::element::Type output_type; + ngraph::opset5::NonMaxSuppression::BoxEncodingType box_encoding; + bool sort_result_descending; + bool is_supported_nms; +}; + + NMSAttributes get_nms4_attrs(const std::shared_ptr& nms4) { + NMSAttributes attrs; + + attrs.box_encoding = ::ngraph::opset5::NonMaxSuppression::BoxEncodingType::CORNER; + attrs.is_supported_nms = true; + attrs.sort_result_descending = true; + attrs.output_type = ::ngraph::element::i64; + + switch (nms4->get_box_encoding()) { + case ::ngraph::opset4::NonMaxSuppression::BoxEncodingType::CENTER: + attrs.box_encoding = ::ngraph::opset5::NonMaxSuppression::BoxEncodingType::CENTER; + break; + case ::ngraph::opset4::NonMaxSuppression::BoxEncodingType::CORNER: + attrs.box_encoding = ::ngraph::opset5::NonMaxSuppression::BoxEncodingType::CORNER; + break; + default: + throw ngraph_error("NonMaxSuppression layer " + nms4->get_friendly_name() + + " has unsupported box encoding"); + } + + attrs.sort_result_descending = nms4->get_sort_result_descending(); + attrs.output_type = nms4->get_output_type(); + + return attrs; + } + + NMSAttributes get_nms3_attrs(const std::shared_ptr& nms3) { + NMSAttributes attrs; + + attrs.box_encoding = ::ngraph::opset5::NonMaxSuppression::BoxEncodingType::CORNER; + attrs.is_supported_nms = true; + attrs.sort_result_descending = true; + attrs.output_type = ::ngraph::element::i64; + + switch (nms3->get_box_encoding()) { + case ::ngraph::opset3::NonMaxSuppression::BoxEncodingType::CENTER: + attrs.box_encoding = ::ngraph::opset5::NonMaxSuppression::BoxEncodingType::CENTER; + break; + case ::ngraph::opset3::NonMaxSuppression::BoxEncodingType::CORNER: + attrs.box_encoding = ::ngraph::opset5::NonMaxSuppression::BoxEncodingType::CORNER; + break; + default: + throw ngraph_error("NonMaxSuppression layer " + nms3->get_friendly_name() + + " has unsupported box encoding"); + } + + attrs.sort_result_descending = nms3->get_sort_result_descending(); + attrs.output_type = nms3->get_output_type(); + + return attrs; + } + + NMSAttributes get_nms1_attrs(const std::shared_ptr& nms1) { + NMSAttributes attrs; + + attrs.box_encoding = ::ngraph::opset5::NonMaxSuppression::BoxEncodingType::CORNER; + attrs.is_supported_nms = true; + attrs.sort_result_descending = true; + attrs.output_type = ::ngraph::element::i64; + + switch (nms1->get_box_encoding()) { + case ::ngraph::opset1::NonMaxSuppression::BoxEncodingType::CENTER: + attrs.box_encoding = ::ngraph::opset5::NonMaxSuppression::BoxEncodingType::CENTER; + break; + case ::ngraph::opset1::NonMaxSuppression::BoxEncodingType::CORNER: + attrs.box_encoding = ::ngraph::opset5::NonMaxSuppression::BoxEncodingType::CORNER; + break; + default: + throw ngraph_error("NonMaxSuppression layer " + nms1->get_friendly_name() + + " has unsupported box encoding"); + } + + attrs.sort_result_descending = nms1->get_sort_result_descending(); + + return attrs; + } + + NMSAttributes get_nms_attrs(const std::shared_ptr& root) { + NMSAttributes attrs; + attrs.output_type = ::ngraph::element::i64; + attrs.box_encoding = ::ngraph::opset5::NonMaxSuppression::BoxEncodingType::CORNER; + attrs.sort_result_descending = false; + attrs.is_supported_nms = false; + + auto nms_4 = std::dynamic_pointer_cast(root); + if (nms_4) { + return get_nms4_attrs(nms_4); + } + auto nms_3 = std::dynamic_pointer_cast(root); + if (nms_3) { + return get_nms3_attrs(nms_3); + } + auto nms_1 = std::dynamic_pointer_cast(root); + if (nms_1) { + return get_nms1_attrs(nms_1); + } + + return attrs; + } + + bool callback_func(pattern::Matcher &m) { + auto root = m.get_match_root(); + + auto attrs = get_nms_attrs(root); + if (!attrs.is_supported_nms) { + return false; + } + + const auto new_args = root->input_values(); + + size_t num_of_args = new_args.size(); + + const auto& arg2 = num_of_args > 2 ? new_args.at(2) : ngraph::opset5::Constant::create(element::i64, Shape{}, {0}); + const auto& arg3 = num_of_args > 3 ? new_args.at(3) : ngraph::opset5::Constant::create(element::f32, Shape{}, {.0f}); + const auto& arg4 = num_of_args > 4 ? new_args.at(4) : ngraph::opset5::Constant::create(element::f32, Shape{}, {.0f}); + + // list of new nGraph operations + std::list> new_ops_list; + + if (num_of_args <= 4) { + new_ops_list.push_front(arg4.get_node_shared_ptr()); + } + if (num_of_args <= 3) { + new_ops_list.push_front(arg3.get_node_shared_ptr()); + } + if (num_of_args <= 2) { + new_ops_list.push_front(arg2.get_node_shared_ptr()); + } + + const auto nms_5 = std::make_shared( + new_args.at(0), + new_args.at(1), + arg2, + arg3, + arg4, + attrs.box_encoding, + attrs.sort_result_descending, + attrs.output_type); + + new_ops_list.push_back(nms_5); + + // vector of new nGraph operations + NodeVector new_ops(new_ops_list.begin(), new_ops_list.end()); + + nms_5->set_friendly_name(root->get_friendly_name()); + ngraph::copy_runtime_info(root, new_ops); + root->output(0).replace(nms_5->output(0)); + return true; + } +} // namespace + +NGRAPH_RTTI_DEFINITION(ngraph::pass::ConvertPreviousNMSToNMS5, "ConvertPreviousNMSToNMS5", 0); + +NGRAPH_RTTI_DEFINITION(ngraph::pass::ConvertNMS4ToNMS5, "ConvertNMS4ToNMS5", 0); + +ngraph::pass::ConvertNMS4ToNMS5::ConvertNMS4ToNMS5() { + auto nms = ngraph::pattern::wrap_type(); + ngraph::matcher_pass_callback callback = callback_func; + + auto m = std::make_shared(nms, "ConvertNMS4ToNMS5"); + this->register_matcher(m, callback); +} + +NGRAPH_RTTI_DEFINITION(ngraph::pass::ConvertNMS3ToNMS5, "ConvertNMS3ToNMS5", 0); + +ngraph::pass::ConvertNMS3ToNMS5::ConvertNMS3ToNMS5() { + auto nms = ngraph::pattern::wrap_type(); + ngraph::matcher_pass_callback callback = callback_func; + + auto m = std::make_shared(nms, "ConvertNMS3ToNMS5"); + this->register_matcher(m, callback); +} + +NGRAPH_RTTI_DEFINITION(ngraph::pass::ConvertNMS1ToNMS5, "ConvertNMS1ToNMS5", 0); + +ngraph::pass::ConvertNMS1ToNMS5::ConvertNMS1ToNMS5() { + auto nms = ngraph::pattern::wrap_type(); + ngraph::matcher_pass_callback callback = callback_func; + + auto m = std::make_shared(nms, "ConvertNMS1ToNMS5"); + this->register_matcher(m, callback); +} diff --git a/inference-engine/src/transformations/src/transformations/opset_conversions/convert_opset3_to_opset2.cpp b/inference-engine/src/transformations/src/transformations/opset_conversions/convert_opset3_to_opset2.cpp index 85f54d1e433512..2ad381e13670cf 100644 --- a/inference-engine/src/transformations/src/transformations/opset_conversions/convert_opset3_to_opset2.cpp +++ b/inference-engine/src/transformations/src/transformations/opset_conversions/convert_opset3_to_opset2.cpp @@ -5,7 +5,6 @@ #include "transformations/opset_conversions/convert_opset3_to_opset2.hpp" #include "transformations/op_conversions/convert_broadcast3.hpp" -#include "transformations/op_conversions/convert_nms3.hpp" #include "transformations/op_conversions/convert_shapeof3.hpp" #include "transformations/op_conversions/convert_shuffle_channels3.hpp" #include "transformations/op_conversions/convert_topk3.hpp" @@ -25,7 +24,6 @@ bool ngraph::pass::ConvertOpSet3ToOpSet2::run_on_function(std::shared_ptr(); - manager.register_pass(); manager.register_pass(); manager.register_pass(); manager.register_pass(); diff --git a/inference-engine/tests/functional/inference_engine/ngraph_reader/non_max_suppression_tests.cpp b/inference-engine/tests/functional/inference_engine/ngraph_reader/non_max_suppression_tests.cpp index bc2f3aa651eee3..27ee39178690ed 100644 --- a/inference-engine/tests/functional/inference_engine/ngraph_reader/non_max_suppression_tests.cpp +++ b/inference-engine/tests/functional/inference_engine/ngraph_reader/non_max_suppression_tests.cpp @@ -4,12 +4,30 @@ #include #include "ngraph_reader_tests.hpp" -/* -TEST_F(NGraphReaderTests, ReadNonMaxSuppression) { + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "common_test_utils/ngraph_test_utils.hpp" +#include "generic_ie.hpp" + +#include "legacy/convert_function_to_cnn_network.hpp" + +using namespace ngraph; + +TEST_F(NGraphReaderTests, ReadNonMaxSuppression5) { std::string model = R"V0G0N( - + @@ -19,7 +37,7 @@ TEST_F(NGraphReaderTests, ReadNonMaxSuppression) { - + @@ -29,55 +47,69 @@ TEST_F(NGraphReaderTests, ReadNonMaxSuppression) { - + - + - + - + - + - + - + + + + + + + - + 1 15130 4 - + 1 80 15130 - - - + + + + - + 15130 3 + + 15130 + 3 + + + 1 + - + - + 15130 3 - + 15130 3 @@ -89,7 +121,55 @@ TEST_F(NGraphReaderTests, ReadNonMaxSuppression) { - + + + + 15130 + 3 + + + + + + + 1 + + + 1 + + + + + 1 + + + + + + + 1 + + + + + + + 15130 + 3 + + + 15130 + 3 + + + + + 15130 + 3 + + + + 15130 @@ -99,34 +179,39 @@ TEST_F(NGraphReaderTests, ReadNonMaxSuppression) { - - - - - - - - + + + + + + + + + + + + + + + )V0G0N"; std::string modelV5 = R"V0G0N( - + - - + - + 1 15130 4 - - + - + 1 80 15130 @@ -134,25 +219,37 @@ TEST_F(NGraphReaderTests, ReadNonMaxSuppression) { - - + + 1 + + + + - - + + 1 + + + + - - + + 1 + + + + - - + + 1 @@ -164,31 +261,80 @@ TEST_F(NGraphReaderTests, ReadNonMaxSuppression) { 80 15130 - - - + + 1 + + + 1 + + + 1 + - - 15130 + + 16000 3 + + 16000 + 3 + + + 1 + - + + - 15130 + 16000 3 - 15130 + 16000 3 - - 15130 + + 16000 + 3 + + + + + + + + 1 + + + 1 + + + + + 1 + + + + + + + + 16000 + 3 + + + 16000 + 3 + + + + + 16000 3 @@ -202,18 +348,167 @@ TEST_F(NGraphReaderTests, ReadNonMaxSuppression) { + + + + )V0G0N"; - compareIRs(model, modelV5, 16, [](Blob::Ptr& weights) { + compareIRs(model, modelV5, 20, [](Blob::Ptr& weights) { auto * i64w = weights->buffer().as(); i64w[0] = 200; auto * fp32w = weights->buffer().as(); fp32w[2] = 0.5; fp32w[3] = 0.05; + fp32w[4] = 0.0; }); } - */ +TEST_F(NGraphReaderTests, ReadNonMaxSuppression4) { + std::string model = R"V0G0N( + + + + + + + 1 + 15130 + 4 + + + + + + + + 1 + 80 + 15130 + + + + + + + + + + + + + + + + + + + + + + + + + + 1 + 15130 + 4 + + + 1 + 80 + 15130 + + + + + + + + 16000 + 3 + + + + + + + 16000 + 3 + + + 16000 + 3 + + + + + 16000 + 3 + + + + + + + 16000 + 3 + + + + + + + + + + + + + + + +)V0G0N"; + + constexpr size_t weightsSize = 16; + + Blob::Ptr weights; + + weights = make_shared_blob(TensorDesc(Precision::U8, {weightsSize}, Layout::C)); + weights->allocate(); + CommonTestUtils::fill_data(weights->buffer().as(), weights->size() / sizeof(float)); + + auto * i64w = weights->buffer().as(); + i64w[0] = 200; + + auto * fp32w = weights->buffer().as(); + fp32w[2] = 0.5; + fp32w[3] = 0.05; + + Core ie; + + auto ngraphImpl = ie.ReadNetwork(model, weights); + auto graph = ngraph::clone_function(*ngraphImpl.getFunction()); + + ::ngraph::pass::Manager manager; + manager.register_pass<::ngraph::pass::InitNodeInfo>(); + manager.register_pass<::ngraph::pass::CommonOptimizations>(); + manager.run_passes(graph); + + auto boxes = std::make_shared(element::f32, Shape{1, 15130, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 80, 15130}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {200}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.5}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.05}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold, + opset5::NonMaxSuppression::BoxEncodingType::CORNER, true, ngraph::element::i32); + + auto mul = std::make_shared(nms, nms); + auto graph_ref = std::make_shared(NodeVector{mul}, ParameterVector{boxes, scores}); + + auto res = compare_functions(graph, graph_ref); + ASSERT_TRUE(res.first) << res.second; +} diff --git a/inference-engine/tests/functional/inference_engine/transformations/convert_nms3_test.cpp b/inference-engine/tests/functional/inference_engine/transformations/convert_nms3_test.cpp deleted file mode 100644 index ae4ee04c26b7c6..00000000000000 --- a/inference-engine/tests/functional/inference_engine/transformations/convert_nms3_test.cpp +++ /dev/null @@ -1,62 +0,0 @@ -// Copyright (C) 2020 Intel Corporation -// SPDX-License-Identifier: Apache-2.0 -// - -#include - -#include -#include -#include - -#include -#include -#include -#include -#include -#include -#include - -#include "common_test_utils/ngraph_test_utils.hpp" - -using namespace testing; -using namespace ngraph; - -TEST(TransformationTests, ConvertNMS3I32Output) { - std::shared_ptr f(nullptr), f_ref(nullptr); - { - auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); - auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); - auto max_output_boxes_per_class = opset3::Constant::create(element::i64, Shape{}, {10}); - auto iou_threshold = opset3::Constant::create(element::f32, Shape{}, {0.75}); - auto score_threshold = opset3::Constant::create(element::f32, Shape{}, {0.7}); - auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, - iou_threshold, score_threshold, opset1::NonMaxSuppression::BoxEncodingType::CORNER, true); - nms->set_friendly_name("nms"); - - f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); - - pass::InitNodeInfo().run_on_function(f); - pass::ConvertNMS1ToNMS3().run_on_function(f); - ASSERT_NO_THROW(check_rt_info(f)); - } - - { - auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); - auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); - auto max_output_boxes_per_class = opset3::Constant::create(element::i64, Shape{}, {10}); - auto iou_threshold = opset3::Constant::create(element::f32, Shape{}, {0.75}); - auto score_threshold = opset3::Constant::create(element::f32, Shape{}, {0.7}); - auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, - iou_threshold, score_threshold, opset3::NonMaxSuppression::BoxEncodingType::CORNER, true); - nms->set_friendly_name("nms"); - - f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); - } - - auto res = compare_functions(f, f_ref); - ASSERT_TRUE(res.first) << res.second; - - auto result_node_of_converted_f = f->get_output_op(0); - auto nms_node = result_node_of_converted_f->input(0).get_source_output().get_node_shared_ptr(); - ASSERT_TRUE(nms_node->get_friendly_name() == "nms") << "Transformation ConvertNMS1ToNMS3 should keep output names.\n"; -} diff --git a/inference-engine/tests/functional/inference_engine/transformations/convert_nms4_test.cpp b/inference-engine/tests/functional/inference_engine/transformations/convert_nms4_test.cpp deleted file mode 100644 index a5eb5b50568351..00000000000000 --- a/inference-engine/tests/functional/inference_engine/transformations/convert_nms4_test.cpp +++ /dev/null @@ -1,150 +0,0 @@ -// Copyright (C) 2020 Intel Corporation -// SPDX-License-Identifier: Apache-2.0 -// - -#include - -#include -#include - -#include -#include -#include -#include -#include -#include -#include - -#include "common_test_utils/ngraph_test_utils.hpp" - -using namespace testing; -using namespace ngraph; - -TEST(TransformationTests, ConvertNMS4ToNMSIEStatic) { - std::shared_ptr f(nullptr), f_ref(nullptr); - { - auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); - auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); - auto max_output_boxes_per_class = opset4::Constant::create(element::i64, Shape{}, {10}); - auto iou_threshold = opset4::Constant::create(element::f32, Shape{}, {0.75}); - auto score_threshold = opset4::Constant::create(element::f32, Shape{}, {0.7}); - auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold, - opset4::NonMaxSuppression::BoxEncodingType::CORNER, true); - - f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); - - const auto &orig_shape = f->get_output_partial_shape(0); - pass::Manager manager; - manager.register_pass(); - manager.register_pass(); - manager.run_passes(f); - ASSERT_NO_THROW(check_rt_info(f)); - ASSERT_TRUE(f->get_output_partial_shape(0).is_static()) << "Shape " << f->get_output_partial_shape(0) << " should be static"; - } - - { - auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); - auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); - auto max_output_boxes_per_class = opset4::Constant::create(element::i64, Shape{1}, {10}); - auto iou_threshold = opset4::Constant::create(element::f32, Shape{}, {0.75}); - auto score_threshold = opset4::Constant::create(element::f32, Shape{}, {0.7}); - auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, - std::make_shared(iou_threshold, - opset4::Constant::create(element::i64, Shape{1}, {0})), - std::make_shared(score_threshold, - opset4::Constant::create(element::i64, Shape{1}, {0})), - 0, true); - nms->set_friendly_name("nms"); - - f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); - ASSERT_TRUE(f_ref->get_output_partial_shape(0).is_static()) << "Shape " << f_ref->get_output_partial_shape(0) << " should be static"; - } - - auto res = compare_functions(f, f_ref); - ASSERT_TRUE(res.first) << res.second; -} - -TEST(TransformationTests, ConvertNMS4ToNMSIEDynamic1) { - std::shared_ptr f(nullptr), f_ref(nullptr); - { - auto boxes = std::make_shared(element::f32, PartialShape::dynamic()); - auto scores = std::make_shared(element::f32, PartialShape::dynamic()); - auto max_output_boxes_per_class = opset4::Constant::create(element::i64, Shape{}, {10}); - auto iou_threshold = opset4::Constant::create(element::f32, Shape{}, {0.75}); - auto score_threshold = opset4::Constant::create(element::f32, Shape{}, {0.7}); - auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold, - opset4::NonMaxSuppression::BoxEncodingType::CORNER, true, element::i64); - - f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); - - pass::Manager manager; - manager.register_pass(); - manager.register_pass(); - manager.run_passes(f); - f->validate_nodes_and_infer_types(); - ASSERT_NO_THROW(check_rt_info(f)); - } - - { - auto boxes = std::make_shared(element::f32, PartialShape::dynamic()); - auto scores = std::make_shared(element::f32, PartialShape::dynamic()); - auto max_output_boxes_per_class = opset4::Constant::create(element::i64, Shape{1}, {10}); - auto iou_threshold = opset4::Constant::create(element::f32, Shape{}, {0.75}); - auto score_threshold = opset4::Constant::create(element::f32, Shape{}, {0.7}); - auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, - std::make_shared(iou_threshold, - opset4::Constant::create(element::i64, Shape{1}, {0})), - std::make_shared(score_threshold, - opset4::Constant::create(element::i64, Shape{1}, {0})), - 0, true); - - f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); - } - - auto res = compare_functions(f, f_ref); - ASSERT_TRUE(res.first) << res.second; -} - -// LPT to nGraph migration: temporary disabling unexpected not reproduced fails on CI: -// https://openvino-ci.intel.com/job/private-ci/job/ie/job/build-linux-ubuntu18_i386/478/ -TEST(TransformationTests, DISABLED_ConvertNMS4ToNMSIEDynamic2) { - std::shared_ptr f(nullptr), f_ref(nullptr); - { - auto boxes = std::make_shared(element::f32, PartialShape{DYN, 1000, 4}); - auto scores = std::make_shared(element::f32, PartialShape{DYN, 1, 1000}); - auto max_output_boxes_per_class = opset4::Constant::create(element::i64, Shape{}, {10}); - auto iou_threshold = opset4::Constant::create(element::f32, Shape{}, {0.75}); - auto score_threshold = opset4::Constant::create(element::f32, Shape{}, {0.7}); - auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold, - opset4::NonMaxSuppression::BoxEncodingType::CORNER, true, element::i32); - - f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); - - pass::Manager manager; - manager.register_pass(); - manager.register_pass(); - manager.run_passes(f); - - f->validate_nodes_and_infer_types(); - ASSERT_NO_THROW(check_rt_info(f)); - } - - { - auto boxes = std::make_shared(element::f32, PartialShape{DYN, 1000, 4}); - auto scores = std::make_shared(element::f32, PartialShape{DYN, 1, 1000}); - auto max_output_boxes_per_class = opset4::Constant::create(element::i64, Shape{1}, {10}); - auto iou_threshold = opset4::Constant::create(element::f32, Shape{}, {0.75}); - auto score_threshold = opset4::Constant::create(element::f32, Shape{}, {0.7}); - auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, - std::make_shared(iou_threshold, - opset4::Constant::create(element::i64, Shape{1}, {0})), - std::make_shared(score_threshold, - opset4::Constant::create(element::i64, Shape{1}, {0})), - 0, true); - - f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); - } - - auto res = compare_functions(f, f_ref); - ASSERT_TRUE(res.first) << res.second; -} diff --git a/inference-engine/tests/functional/inference_engine/transformations/convert_nms5_test.cpp b/inference-engine/tests/functional/inference_engine/transformations/convert_nms5_test.cpp new file mode 100644 index 00000000000000..6ba473952a4bf4 --- /dev/null +++ b/inference-engine/tests/functional/inference_engine/transformations/convert_nms5_test.cpp @@ -0,0 +1,763 @@ +// Copyright (C) 2020 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include + +#include +#include + +#include +#include +#include +#include +#include +#include +#include + +#include "common_test_utils/ngraph_test_utils.hpp" + +using namespace testing; +using namespace ngraph; + +TEST(TransformationTests, ConvertNMS5ToNMSIEStaticSixInputs) { + std::shared_ptr f(nullptr), f_ref(nullptr); + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.75}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.7}); + auto soft_nms_sigma = ngraph::opset5::Constant::create(element::f32, Shape{}, {0.25}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold, + soft_nms_sigma, opset5::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + + pass::Manager manager; + manager.register_pass(); + manager.register_pass(); + manager.run_passes(f); + ASSERT_NO_THROW(check_rt_info(f)); + ASSERT_TRUE(f->get_output_partial_shape(0).same_scheme(PartialShape{Dimension::dynamic(), 3})); + } + + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.75}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.7}); + auto soft_nms_sigma = ngraph::opset5::Constant::create(element::f32, Shape{}, {0.25}); + + auto one_dim_shape = Shape{1}; + auto new_max_per_class = std::make_shared(max_output_boxes_per_class, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_iou_threshold = std::make_shared(iou_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_score_threshold = std::make_shared(score_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_soft_nms_sigma = std::make_shared(soft_nms_sigma, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto nms = std::make_shared(boxes, scores, new_max_per_class, new_iou_threshold, new_score_threshold, + new_soft_nms_sigma, 0, true); + nms->set_friendly_name("nms"); + + f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + ASSERT_TRUE(f_ref->get_output_partial_shape(0).same_scheme(PartialShape{Dimension::dynamic(), 3})); + } + + auto res = compare_functions(f, f_ref); + ASSERT_TRUE(res.first) << res.second; +} + +TEST(TransformationTests, ConvertNMS5ToNMSIEStaticFiveInputs) { + std::shared_ptr f(nullptr), f_ref(nullptr); + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.75}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.7}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold, + opset5::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + + pass::Manager manager; + manager.register_pass(); + manager.register_pass(); + manager.run_passes(f); + ASSERT_NO_THROW(check_rt_info(f)); + ASSERT_TRUE(f->get_output_partial_shape(0).same_scheme(PartialShape{Dimension::dynamic(), 3})); + } + + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.75}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.7}); + + auto one_dim_shape = Shape{1}; + auto new_max_per_class = std::make_shared(max_output_boxes_per_class, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_iou_threshold = std::make_shared(iou_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_score_threshold = std::make_shared(score_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto nms = std::make_shared(boxes, scores, new_max_per_class, new_iou_threshold, new_score_threshold, + 0, true); + nms->set_friendly_name("nms"); + + f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + ASSERT_TRUE(f_ref->get_output_partial_shape(0).same_scheme(PartialShape{Dimension::dynamic(), 3})); + } + + auto res = compare_functions(f, f_ref); + ASSERT_TRUE(res.first) << res.second; +} + +TEST(TransformationTests, ConvertNMS5ToNMSIEStaticFourInputs) { + std::shared_ptr f(nullptr), f_ref(nullptr); + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.75}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, + opset5::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + + pass::Manager manager; + manager.register_pass(); + manager.register_pass(); + manager.run_passes(f); + ASSERT_NO_THROW(check_rt_info(f)); + ASSERT_TRUE(f->get_output_partial_shape(0).same_scheme(PartialShape{Dimension::dynamic(), 3})); + } + + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.75}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.0f}); + + auto one_dim_shape = Shape{1}; + auto new_max_per_class = std::make_shared(max_output_boxes_per_class, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_iou_threshold = std::make_shared(iou_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_score_threshold = std::make_shared(score_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto nms = std::make_shared(boxes, scores, new_max_per_class, new_iou_threshold, new_score_threshold, + 0, true); + nms->set_friendly_name("nms"); + + f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + ASSERT_TRUE(f_ref->get_output_partial_shape(0).same_scheme(PartialShape{Dimension::dynamic(), 3})); + } + + auto res = compare_functions(f, f_ref); + ASSERT_TRUE(res.first) << res.second; +} + +TEST(TransformationTests, ConvertNMS5ToNMSIEStaticThreeInputs) { + std::shared_ptr f(nullptr), f_ref(nullptr); + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, + opset5::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + + pass::Manager manager; + manager.register_pass(); + manager.register_pass(); + manager.run_passes(f); + ASSERT_NO_THROW(check_rt_info(f)); + ASSERT_TRUE(f->get_output_partial_shape(0).same_scheme(PartialShape{Dimension::dynamic(), 3})); + } + + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.0f}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.0f}); + + auto one_dim_shape = Shape{1}; + auto new_max_per_class = std::make_shared(max_output_boxes_per_class, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_iou_threshold = std::make_shared(iou_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_score_threshold = std::make_shared(score_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto nms = std::make_shared(boxes, scores, new_max_per_class, new_iou_threshold, new_score_threshold, + 0, true); + nms->set_friendly_name("nms"); + + f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + ASSERT_TRUE(f_ref->get_output_partial_shape(0).same_scheme(PartialShape{Dimension::dynamic(), 3})); + } + + auto res = compare_functions(f, f_ref); + ASSERT_TRUE(res.first) << res.second; +} + +TEST(TransformationTests, ConvertNMS5ToNMSIEStaticTwoInputs) { + std::shared_ptr f(nullptr), f_ref(nullptr); + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto nms = std::make_shared(boxes, scores, + opset5::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + + pass::Manager manager; + manager.register_pass(); + manager.register_pass(); + manager.run_passes(f); + ASSERT_NO_THROW(check_rt_info(f)); + ASSERT_TRUE(f->get_output_partial_shape(0).same_scheme(PartialShape{Dimension::dynamic(), 3})); + } + + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i32, Shape{}, {0}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.0f}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.0f}); + + auto one_dim_shape = Shape{1}; + auto new_max_per_class = std::make_shared(max_output_boxes_per_class, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_iou_threshold = std::make_shared(iou_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_score_threshold = std::make_shared(score_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto nms = std::make_shared(boxes, scores, new_max_per_class, new_iou_threshold, new_score_threshold, + 0, true); + nms->set_friendly_name("nms"); + + f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + ASSERT_TRUE(f_ref->get_output_partial_shape(0).same_scheme(PartialShape{Dimension::dynamic(), 3})); + } + + auto res = compare_functions(f, f_ref); + ASSERT_TRUE(res.first) << res.second; +} + +TEST(TransformationTests, ConvertNMS5ToNMSIEDynamic1SixInputs) { + std::shared_ptr f(nullptr), f_ref(nullptr); + { + auto boxes = std::make_shared(element::f32, PartialShape::dynamic()); + auto scores = std::make_shared(element::f32, PartialShape::dynamic()); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.75}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.7}); + auto soft_nms_sigma = ngraph::opset5::Constant::create(element::f32, Shape{}, {0.25}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold, + soft_nms_sigma, opset5::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + + const auto &orig_selected_indices_shape = f->get_output_partial_shape(0); + pass::Manager manager; + manager.register_pass(); + manager.register_pass(); + manager.run_passes(f); + f->validate_nodes_and_infer_types(); + ASSERT_NO_THROW(check_rt_info(f)); + } + + { + auto boxes = std::make_shared(element::f32, PartialShape::dynamic()); + auto scores = std::make_shared(element::f32, PartialShape::dynamic()); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.75}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.7}); + auto soft_nms_sigma = ngraph::opset5::Constant::create(element::f32, Shape{}, {0.25}); + + auto one_dim_shape = Shape{1}; + auto new_max_per_class = std::make_shared(max_output_boxes_per_class, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_iou_threshold = std::make_shared(iou_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_score_threshold = std::make_shared(score_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_soft_nms_sigma = std::make_shared(soft_nms_sigma, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto nms = std::make_shared(boxes, scores, new_max_per_class, new_iou_threshold, new_score_threshold, + new_soft_nms_sigma, 0, true); + nms->set_friendly_name("nms"); + + f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + } + + auto res = compare_functions(f, f_ref); + ASSERT_TRUE(res.first) << res.second; +} + +TEST(TransformationTests, ConvertNMS5ToNMSIEDynamic1FiveInputs) { + std::shared_ptr f(nullptr), f_ref(nullptr); + { + auto boxes = std::make_shared(element::f32, PartialShape::dynamic()); + auto scores = std::make_shared(element::f32, PartialShape::dynamic()); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.75}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.7}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold, + opset5::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + + const auto &orig_selected_indices_shape = f->get_output_partial_shape(0); + pass::Manager manager; + manager.register_pass(); + manager.register_pass(); + manager.run_passes(f); + f->validate_nodes_and_infer_types(); + ASSERT_NO_THROW(check_rt_info(f)); + } + + { + auto boxes = std::make_shared(element::f32, PartialShape::dynamic()); + auto scores = std::make_shared(element::f32, PartialShape::dynamic()); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.75}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.7}); + + auto one_dim_shape = Shape{1}; + auto new_max_per_class = std::make_shared(max_output_boxes_per_class, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_iou_threshold = std::make_shared(iou_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_score_threshold = std::make_shared(score_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto nms = std::make_shared(boxes, scores, new_max_per_class, new_iou_threshold, new_score_threshold, + 0, true); + nms->set_friendly_name("nms"); + + f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + } + + auto res = compare_functions(f, f_ref); + ASSERT_TRUE(res.first) << res.second; +} + +TEST(TransformationTests, ConvertNMS5ToNMSIEDynamic1FourInputs) { + std::shared_ptr f(nullptr), f_ref(nullptr); + { + auto boxes = std::make_shared(element::f32, PartialShape::dynamic()); + auto scores = std::make_shared(element::f32, PartialShape::dynamic()); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.75}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, + opset5::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + + const auto &orig_selected_indices_shape = f->get_output_partial_shape(0); + pass::Manager manager; + manager.register_pass(); + manager.register_pass(); + manager.run_passes(f); + f->validate_nodes_and_infer_types(); + ASSERT_NO_THROW(check_rt_info(f)); + } + + { + auto boxes = std::make_shared(element::f32, PartialShape::dynamic()); + auto scores = std::make_shared(element::f32, PartialShape::dynamic()); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.75}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.0f}); + + auto one_dim_shape = Shape{1}; + auto new_max_per_class = std::make_shared(max_output_boxes_per_class, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_iou_threshold = std::make_shared(iou_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_score_threshold = std::make_shared(score_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto nms = std::make_shared(boxes, scores, new_max_per_class, new_iou_threshold, new_score_threshold, + 0, true); + nms->set_friendly_name("nms"); + + f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + } + + auto res = compare_functions(f, f_ref); + ASSERT_TRUE(res.first) << res.second; +} + +TEST(TransformationTests, ConvertNMS5ToNMSIEDynamic1ThreeInputs) { + std::shared_ptr f(nullptr), f_ref(nullptr); + { + auto boxes = std::make_shared(element::f32, PartialShape::dynamic()); + auto scores = std::make_shared(element::f32, PartialShape::dynamic()); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, + opset5::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + + const auto &orig_selected_indices_shape = f->get_output_partial_shape(0); + pass::Manager manager; + manager.register_pass(); + manager.register_pass(); + manager.run_passes(f); + f->validate_nodes_and_infer_types(); + ASSERT_NO_THROW(check_rt_info(f)); + } + + { + auto boxes = std::make_shared(element::f32, PartialShape::dynamic()); + auto scores = std::make_shared(element::f32, PartialShape::dynamic()); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.0f}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.0f}); + + auto one_dim_shape = Shape{1}; + auto new_max_per_class = std::make_shared(max_output_boxes_per_class, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_iou_threshold = std::make_shared(iou_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_score_threshold = std::make_shared(score_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto nms = std::make_shared(boxes, scores, new_max_per_class, new_iou_threshold, new_score_threshold, + 0, true); + nms->set_friendly_name("nms"); + + f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + } + + auto res = compare_functions(f, f_ref); + ASSERT_TRUE(res.first) << res.second; +} + +TEST(TransformationTests, ConvertNMS5ToNMSIEDynamic1TwoInputs) { + std::shared_ptr f(nullptr), f_ref(nullptr); + { + auto boxes = std::make_shared(element::f32, PartialShape::dynamic()); + auto scores = std::make_shared(element::f32, PartialShape::dynamic()); + auto nms = std::make_shared(boxes, scores, + opset5::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + + const auto &orig_selected_indices_shape = f->get_output_partial_shape(0); + pass::Manager manager; + manager.register_pass(); + manager.register_pass(); + manager.run_passes(f); + f->validate_nodes_and_infer_types(); + ASSERT_NO_THROW(check_rt_info(f)); + } + + { + auto boxes = std::make_shared(element::f32, PartialShape::dynamic()); + auto scores = std::make_shared(element::f32, PartialShape::dynamic()); + auto max_output_boxes_per_class = opset5::Constant::create(element::i32, Shape{}, {0}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.0f}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.0f}); + + auto one_dim_shape = Shape{1}; + auto new_max_per_class = std::make_shared(max_output_boxes_per_class, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_iou_threshold = std::make_shared(iou_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_score_threshold = std::make_shared(score_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto nms = std::make_shared(boxes, scores, new_max_per_class, new_iou_threshold, new_score_threshold, + 0, true); + nms->set_friendly_name("nms"); + + f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + } + + auto res = compare_functions(f, f_ref); + ASSERT_TRUE(res.first) << res.second; +} + +TEST(TransformationTests, ConvertNMS5ToNMSIEDynamic2SixInputs) { + std::shared_ptr f(nullptr), f_ref(nullptr); + { + auto boxes = std::make_shared(element::f32, PartialShape{DYN, 1000, 4}); + auto scores = std::make_shared(element::f32, PartialShape{DYN, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.75}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.7}); + auto soft_nms_sigma = ngraph::opset5::Constant::create(element::f32, Shape{}, {0.25}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold, + soft_nms_sigma, opset5::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + + pass::Manager manager; + manager.register_pass(); + manager.register_pass(); + manager.run_passes(f); + f->validate_nodes_and_infer_types(); + ASSERT_NO_THROW(check_rt_info(f)); + } + + { + auto boxes = std::make_shared(element::f32, PartialShape{DYN, 1000, 4}); + auto scores = std::make_shared(element::f32, PartialShape{DYN, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.75}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.7}); + auto soft_nms_sigma = ngraph::opset5::Constant::create(element::f32, Shape{}, {0.25}); + + auto one_dim_shape = Shape{1}; + auto new_max_per_class = std::make_shared(max_output_boxes_per_class, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_iou_threshold = std::make_shared(iou_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_score_threshold = std::make_shared(score_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_soft_nms_sigma = std::make_shared(soft_nms_sigma, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto nms = std::make_shared(boxes, scores, new_max_per_class, new_iou_threshold, new_score_threshold, + new_soft_nms_sigma, 0, true); + nms->set_friendly_name("nms"); + + f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + } + + auto res = compare_functions(f, f_ref); + ASSERT_TRUE(res.first) << res.second; +} + +TEST(TransformationTests, ConvertNMS5ToNMSIEDynamic2FiveInputs) { + std::shared_ptr f(nullptr), f_ref(nullptr); + { + auto boxes = std::make_shared(element::f32, PartialShape{DYN, 1000, 4}); + auto scores = std::make_shared(element::f32, PartialShape{DYN, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.75}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.7}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold, + opset5::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + + pass::Manager manager; + manager.register_pass(); + manager.register_pass(); + manager.run_passes(f); + f->validate_nodes_and_infer_types(); + ASSERT_NO_THROW(check_rt_info(f)); + } + + { + auto boxes = std::make_shared(element::f32, PartialShape{DYN, 1000, 4}); + auto scores = std::make_shared(element::f32, PartialShape{DYN, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.75}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.7}); + + auto one_dim_shape = Shape{1}; + auto new_max_per_class = std::make_shared(max_output_boxes_per_class, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_iou_threshold = std::make_shared(iou_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_score_threshold = std::make_shared(score_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto nms = std::make_shared(boxes, scores, new_max_per_class, new_iou_threshold, new_score_threshold, + 0, true); + nms->set_friendly_name("nms"); + + f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + } + + auto res = compare_functions(f, f_ref); + ASSERT_TRUE(res.first) << res.second; +} + +TEST(TransformationTests, ConvertNMS5ToNMSIEDynamic2FourInputs) { + std::shared_ptr f(nullptr), f_ref(nullptr); + { + auto boxes = std::make_shared(element::f32, PartialShape{DYN, 1000, 4}); + auto scores = std::make_shared(element::f32, PartialShape{DYN, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.75}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, + opset5::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + + pass::Manager manager; + manager.register_pass(); + manager.register_pass(); + manager.run_passes(f); + f->validate_nodes_and_infer_types(); + ASSERT_NO_THROW(check_rt_info(f)); + } + + { + auto boxes = std::make_shared(element::f32, PartialShape{DYN, 1000, 4}); + auto scores = std::make_shared(element::f32, PartialShape{DYN, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.75}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.0f}); + + auto one_dim_shape = Shape{1}; + auto new_max_per_class = std::make_shared(max_output_boxes_per_class, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_iou_threshold = std::make_shared(iou_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_score_threshold = std::make_shared(score_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto nms = std::make_shared(boxes, scores, new_max_per_class, new_iou_threshold, new_score_threshold, + 0, true); + nms->set_friendly_name("nms"); + + f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + } + + auto res = compare_functions(f, f_ref); + ASSERT_TRUE(res.first) << res.second; +} + +TEST(TransformationTests, ConvertNMS5ToNMSIEDynamic2ThreeInputs) { + std::shared_ptr f(nullptr), f_ref(nullptr); + { + auto boxes = std::make_shared(element::f32, PartialShape{DYN, 1000, 4}); + auto scores = std::make_shared(element::f32, PartialShape{DYN, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, + opset5::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + + pass::Manager manager; + manager.register_pass(); + manager.register_pass(); + manager.run_passes(f); + f->validate_nodes_and_infer_types(); + ASSERT_NO_THROW(check_rt_info(f)); + } + + { + auto boxes = std::make_shared(element::f32, PartialShape{DYN, 1000, 4}); + auto scores = std::make_shared(element::f32, PartialShape{DYN, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.0f}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.0f}); + + auto one_dim_shape = Shape{1}; + auto new_max_per_class = std::make_shared(max_output_boxes_per_class, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_iou_threshold = std::make_shared(iou_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_score_threshold = std::make_shared(score_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto nms = std::make_shared(boxes, scores, new_max_per_class, new_iou_threshold, new_score_threshold, + 0, true); + nms->set_friendly_name("nms"); + + f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + } + + auto res = compare_functions(f, f_ref); + ASSERT_TRUE(res.first) << res.second; +} + +TEST(TransformationTests, ConvertNMS5ToNMSIEDynamic2TwoInputs) { + std::shared_ptr f(nullptr), f_ref(nullptr); + { + auto boxes = std::make_shared(element::f32, PartialShape{DYN, 1000, 4}); + auto scores = std::make_shared(element::f32, PartialShape{DYN, 1, 1000}); + auto nms = std::make_shared(boxes, scores, + opset5::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + + pass::Manager manager; + manager.register_pass(); + manager.register_pass(); + manager.run_passes(f); + f->validate_nodes_and_infer_types(); + ASSERT_NO_THROW(check_rt_info(f)); + } + + { + auto boxes = std::make_shared(element::f32, PartialShape{DYN, 1000, 4}); + auto scores = std::make_shared(element::f32, PartialShape{DYN, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i32, Shape{}, {0}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.0f}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.0f}); + + auto one_dim_shape = Shape{1}; + auto new_max_per_class = std::make_shared(max_output_boxes_per_class, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_iou_threshold = std::make_shared(iou_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto new_score_threshold = std::make_shared(score_threshold, + opset5::Constant::create(ngraph::element::i64, one_dim_shape, + one_dim_shape), true); + auto nms = std::make_shared(boxes, scores, new_max_per_class, new_iou_threshold, new_score_threshold, + 0, true); + nms->set_friendly_name("nms"); + + f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + } + + auto res = compare_functions(f, f_ref); + ASSERT_TRUE(res.first) << res.second; +} diff --git a/inference-engine/tests/functional/inference_engine/transformations/convert_previous_nms_to_nms_5.cpp b/inference-engine/tests/functional/inference_engine/transformations/convert_previous_nms_to_nms_5.cpp new file mode 100644 index 00000000000000..a1120f41c419b4 --- /dev/null +++ b/inference-engine/tests/functional/inference_engine/transformations/convert_previous_nms_to_nms_5.cpp @@ -0,0 +1,228 @@ +// Copyright (C) 2020 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include + +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "common_test_utils/ngraph_test_utils.hpp" + +using namespace testing; +using namespace ngraph; + +TEST(TransformationTests, ConvertNMS4FiveInputsToNMS5) { + std::shared_ptr f(nullptr), f_ref(nullptr); + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto max_output_boxes_per_class = opset4::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset4::Constant::create(element::f32, Shape{}, {0.75}); + auto score_threshold = opset4::Constant::create(element::f32, Shape{}, {0.7}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold, + opset4::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + + pass::Manager m; + m.register_pass(); + m.register_pass(); + m.run_passes(f); + ASSERT_NO_THROW(check_rt_info(f)); + } + + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.75}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.7}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold, + opset5::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + } + + auto res = compare_functions(f, f_ref); + ASSERT_TRUE(res.first) << res.second; +} + +TEST(TransformationTests, ConvertNMS4TwoInputsToNMS5) { + std::shared_ptr f(nullptr), f_ref(nullptr); + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto nms = std::make_shared(boxes, scores, opset4::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + + pass::Manager m; + m.register_pass(); + m.register_pass(); + m.run_passes(f); + ASSERT_NO_THROW(check_rt_info(f)); + } + + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {0}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.0f}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.0f}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold, + opset5::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + } + + auto res = compare_functions(f, f_ref); + ASSERT_TRUE(res.first) << res.second; +} + +TEST(TransformationTests, ConvertNMS3FiveInputsToNMS5) { + std::shared_ptr f(nullptr), f_ref(nullptr); + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto max_output_boxes_per_class = opset3::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset3::Constant::create(element::f32, Shape{}, {0.75}); + auto score_threshold = opset3::Constant::create(element::f32, Shape{}, {0.7}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold, + opset3::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + + pass::Manager m; + m.register_pass(); + m.register_pass(); + m.run_passes(f); + ASSERT_NO_THROW(check_rt_info(f)); + } + + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.75}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.7}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold, + opset5::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + } + + auto res = compare_functions(f, f_ref); + ASSERT_TRUE(res.first) << res.second; +} + +TEST(TransformationTests, ConvertNMS3TwoInputsToNMS5) { + std::shared_ptr f(nullptr), f_ref(nullptr); + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto nms = std::make_shared(boxes, scores, opset3::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + + pass::Manager m; + m.register_pass(); + m.register_pass(); + m.run_passes(f); + ASSERT_NO_THROW(check_rt_info(f)); + } + + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {0}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.0f}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.0f}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold, + opset5::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + } + + auto res = compare_functions(f, f_ref); + ASSERT_TRUE(res.first) << res.second; +} + +TEST(TransformationTests, ConvertNMS1FiveInputsToNMS5) { + std::shared_ptr f(nullptr), f_ref(nullptr); + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto max_output_boxes_per_class = opset1::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset1::Constant::create(element::f32, Shape{}, {0.75}); + auto score_threshold = opset1::Constant::create(element::f32, Shape{}, {0.7}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold, + opset1::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + + pass::Manager m; + m.register_pass(); + m.register_pass(); + m.run_passes(f); + ASSERT_NO_THROW(check_rt_info(f)); + } + + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {10}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.75}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.7}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold, + opset5::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + } + + auto res = compare_functions(f, f_ref); + ASSERT_TRUE(res.first) << res.second; +} + +TEST(TransformationTests, ConvertNMS1TwoInputsToNMS5) { + std::shared_ptr f(nullptr), f_ref(nullptr); + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto nms = std::make_shared(boxes, scores, opset1::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + + pass::Manager m; + m.register_pass(); + m.register_pass(); + m.run_passes(f); + ASSERT_NO_THROW(check_rt_info(f)); + } + + { + auto boxes = std::make_shared(element::f32, Shape{1, 1000, 4}); + auto scores = std::make_shared(element::f32, Shape{1, 1, 1000}); + auto max_output_boxes_per_class = opset5::Constant::create(element::i64, Shape{}, {0}); + auto iou_threshold = opset5::Constant::create(element::f32, Shape{}, {0.0f}); + auto score_threshold = opset5::Constant::create(element::f32, Shape{}, {0.0f}); + auto nms = std::make_shared(boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold, + opset5::NonMaxSuppression::BoxEncodingType::CORNER, true); + + f_ref = std::make_shared(NodeVector{nms}, ParameterVector{boxes, scores}); + } + + auto res = compare_functions(f, f_ref); + ASSERT_TRUE(res.first) << res.second; +} diff --git a/ngraph/core/include/ngraph/op/non_max_suppression.hpp b/ngraph/core/include/ngraph/op/non_max_suppression.hpp index c4cc697510dbe7..b6a93610f62a3d 100644 --- a/ngraph/core/include/ngraph/op/non_max_suppression.hpp +++ b/ngraph/core/include/ngraph/op/non_max_suppression.hpp @@ -374,15 +374,16 @@ namespace ngraph } using Node::set_output_type; + int64_t max_boxes_output_from_input() const; + float iou_threshold_from_input() const; + float score_threshold_from_input() const; + float soft_nms_sigma_from_input() const; + protected: BoxEncodingType m_box_encoding = BoxEncodingType::CORNER; bool m_sort_result_descending = true; ngraph::element::Type m_output_type = ngraph::element::i64; void validate(); - int64_t max_boxes_output_from_input() const; - float iou_threshold_from_input() const; - float score_threshold_from_input() const; - float soft_nms_sigma_from_input() const; }; } // namespace v5 } // namespace op @@ -443,4 +444,4 @@ namespace ngraph "AttributeAdapter", 1}; const DiscreteTypeInfo& get_type_info() const override { return type_info; } }; -} // namespace ngraph \ No newline at end of file +} // namespace ngraph diff --git a/ngraph/core/reference/include/ngraph/runtime/reference/non_max_suppression.hpp b/ngraph/core/reference/include/ngraph/runtime/reference/non_max_suppression.hpp new file mode 100644 index 00000000000000..db343abd46dce3 --- /dev/null +++ b/ngraph/core/reference/include/ngraph/runtime/reference/non_max_suppression.hpp @@ -0,0 +1,81 @@ +//***************************************************************************** +// Copyright 2017-2020 Intel Corporation +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +//***************************************************************************** + +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "ngraph/node.hpp" +#include "ngraph/op/util/op_types.hpp" +#include "ngraph/ops.hpp" +#include "ngraph/shape_util.hpp" + +namespace ngraph +{ + namespace runtime + { + namespace reference + { + struct InfoForNMS5 + { + int64_t max_output_boxes_per_class; + float iou_threshold; + float score_threshold; + float soft_nms_sigma; + Shape out_shape; + Shape boxes_shape; + Shape scores_shape; + std::vector boxes_data; + std::vector scores_data; + size_t out_shape_size; + bool sort_result_descending; + ngraph::element::Type output_type; + }; + + InfoForNMS5 get_info_for_nms5_evaluation(const op::v5::NonMaxSuppression* nms5, + const HostTensorVector& inputs); + + void non_max_suppression(const float* boxes_data, + const Shape& boxes_data_shape, + const float* scores_data, + const Shape& scores_data_shape, + int64_t max_output_boxes_per_class, + float iou_threshold, + float score_threshold, + float soft_nms_sigma, + int64_t* selected_indices, + const Shape& selected_indices_shape, + float* selected_scores, + const Shape& selected_scores_shape, + int64_t* valid_outputs, + const bool sort_result_descending); + + void nms5_postprocessing(const HostTensorVector& outputs, + const ngraph::element::Type output_type, + const std::vector& selected_indices, + const std::vector& selected_scores, + int64_t valid_outputs, + const ngraph::element::Type selected_scores_type); + } + } +} diff --git a/ngraph/core/reference/src/runtime/reference/non_max_suppression.cpp b/ngraph/core/reference/src/runtime/reference/non_max_suppression.cpp new file mode 100644 index 00000000000000..44fafd726afa43 --- /dev/null +++ b/ngraph/core/reference/src/runtime/reference/non_max_suppression.cpp @@ -0,0 +1,574 @@ +//***************************************************************************** +// Copyright 2017-2020 Intel Corporation +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +//***************************************************************************** + +#include "ngraph/op/non_max_suppression.hpp" +#include +#include +#include +#include +#include "ngraph/runtime/reference/non_max_suppression.hpp" +#include "ngraph/shape.hpp" + +using namespace ngraph; +using namespace ngraph::runtime::reference; + +struct Rectangle +{ + Rectangle(float y_left, float x_left, float y_right, float x_right) + : y1{y_left} + , x1{x_left} + , y2{y_right} + , x2{x_right} + { + } + + Rectangle() = default; + + float y1 = 0.0f; + float x1 = 0.0f; + float y2 = 0.f; + float x2 = 0.0f; +}; + +static float intersectionOverUnion(const Rectangle& boxI, const Rectangle& boxJ) +{ + float areaI = (boxI.y2 - boxI.y1) * (boxI.x2 - boxI.x1); + float areaJ = (boxJ.y2 - boxJ.y1) * (boxJ.x2 - boxJ.x1); + + if (areaI <= 0.0f || areaJ <= 0.0f) + { + return 0.0f; + } + + float intersection_ymin = std::max(boxI.y1, boxJ.y1); + float intersection_xmin = std::max(boxI.x1, boxJ.x1); + float intersection_ymax = std::min(boxI.y2, boxJ.y2); + float intersection_xmax = std::min(boxI.x2, boxJ.x2); + + float intersection_area = std::max(intersection_ymax - intersection_ymin, 0.0f) * + std::max(intersection_xmax - intersection_xmin, 0.0f); + + return intersection_area / (areaI + areaJ - intersection_area); +} + +struct SelectedIndex +{ + SelectedIndex(int64_t batch_idx, int64_t class_idx, int64_t box_idx) + : batch_index(batch_idx) + , class_index(class_idx) + , box_index(box_idx) + { + } + + SelectedIndex() = default; + + int64_t batch_index = 0; + int64_t class_index = 0; + int64_t box_index = 0; +}; + +struct SelectedScore +{ + SelectedScore(float batch_idx, float class_idx, float score) + : batch_index{batch_idx} + , class_index{class_idx} + , box_score{score} + { + } + + SelectedScore() = default; + + float batch_index = 0.0f; + float class_index = 0.0f; + float box_score = 0.0f; +}; + +struct BoxInfo +{ + BoxInfo(const Rectangle& r, + int64_t idx, + float sc, + int64_t suppress_idx, + int64_t batch_idx, + int64_t class_idx) + : box{r} + , index{idx} + , suppress_begin_index{suppress_idx} + , batch_index{batch_idx} + , class_index{class_idx} + , score{sc} + { + } + + BoxInfo() = default; + + inline bool operator<(const BoxInfo& rhs) const + { + return score < rhs.score || (score == rhs.score && index > rhs.index); + } + + Rectangle box; + int64_t index = 0; + int64_t suppress_begin_index = 0; + int64_t batch_index = 0; + int64_t class_index = 0; + float score = 0.0f; +}; + +using V5BoxEncoding = op::v5::NonMaxSuppression::BoxEncodingType; + +namespace +{ + constexpr size_t boxes_port = 0; + constexpr size_t scores_port = 1; + constexpr size_t max_output_boxes_port = 2; + constexpr size_t iou_threshold_port = 3; + constexpr size_t score_threshold_port = 4; + constexpr size_t soft_nms_sigma_port = 5; + + PartialShape + infer_selected_indices_shape(const std::vector>& inputs, + int64_t max_output_boxes_per_class) + { + const auto boxes_ps = inputs[boxes_port]->get_partial_shape(); + const auto scores_ps = inputs[scores_port]->get_partial_shape(); + + // NonMaxSuppression produces triplets + // that have the following format: [batch_index, class_index, box_index] + PartialShape result = {Dimension::dynamic(), 3}; + + if (boxes_ps.rank().is_static() && scores_ps.rank().is_static()) + { + const auto num_boxes_boxes = boxes_ps[1]; + if (num_boxes_boxes.is_static() && scores_ps[0].is_static() && scores_ps[1].is_static()) + { + const auto num_boxes = num_boxes_boxes.get_length(); + const auto num_classes = scores_ps[1].get_length(); + + result[0] = std::min(num_boxes, max_output_boxes_per_class) * num_classes * + scores_ps[0].get_length(); + } + } + + return result; + } + + void normalize_corner(float* boxes, const Shape& boxes_shape) + { + size_t total_num_of_boxes = shape_size(boxes_shape) / 4; + for (size_t i = 0; i < total_num_of_boxes; ++i) + { + float* current_box = boxes + 4 * i; + + float y1 = current_box[0]; + float x1 = current_box[1]; + float y2 = current_box[2]; + float x2 = current_box[3]; + + float ymin = std::min(y1, y2); + float ymax = std::max(y1, y2); + float xmin = std::min(x1, x2); + float xmax = std::max(x1, x2); + + current_box[0] = ymin; + current_box[1] = xmin; + current_box[2] = ymax; + current_box[3] = xmax; + } + } + + void normalize_center(float* boxes, const Shape& boxes_shape) + { + size_t total_num_of_boxes = shape_size(boxes_shape) / 4; + for (size_t i = 0; i < total_num_of_boxes; ++i) + { + float* current_box = boxes + 4 * i; + + float x_center = current_box[0]; + float y_center = current_box[1]; + float width = current_box[2]; + float height = current_box[3]; + + float y1 = y_center - height / 2.0; + float x1 = x_center - width / 2.0; + float y2 = y_center + height / 2.0; + float x2 = x_center + width / 2.0; + + current_box[0] = y1; + current_box[1] = x1; + current_box[2] = y2; + current_box[3] = x2; + } + } + + void normalize_box_encoding(float* boxes, + const Shape& boxes_shape, + const V5BoxEncoding box_encoding) + { + if (box_encoding == V5BoxEncoding::CORNER) + { + normalize_corner(boxes, boxes_shape); + } + else + { + normalize_center(boxes, boxes_shape); + } + } + + std::vector get_floats(const std::shared_ptr& input, const Shape& shape) + { + size_t input_size = shape_size(shape); + std::vector result(input_size); + + switch (input->get_element_type()) + { + case element::Type_t::bf16: + { + bfloat16* p = input->get_data_ptr(); + for (size_t i = 0; i < input_size; ++i) + { + result[i] = float(p[i]); + } + } + break; + case element::Type_t::f16: + { + float16* p = input->get_data_ptr(); + for (size_t i = 0; i < input_size; ++i) + { + result[i] = float(p[i]); + } + } + break; + case element::Type_t::f32: + { + float* p = input->get_data_ptr(); + memcpy(result.data(), p, input_size * sizeof(float)); + } + break; + default: throw std::runtime_error("Unsupported data type in op NonMaxSuppression-5"); break; + } + + return result; + } + + std::vector prepare_boxes_data(const std::shared_ptr& boxes, + const Shape& boxes_shape, + const V5BoxEncoding box_encoding) + { + auto result = get_floats(boxes, boxes_shape); + normalize_box_encoding(result.data(), boxes_shape, box_encoding); + return result; + } + + std::vector prepare_scores_data(const std::shared_ptr& scores, + const Shape& scores_shape) + { + auto result = get_floats(scores, scores_shape); + return result; + } +} + +namespace ngraph +{ + namespace runtime + { + namespace reference + { + InfoForNMS5 get_info_for_nms5_evaluation(const op::v5::NonMaxSuppression* nms5, + const HostTensorVector& inputs) + { + InfoForNMS5 result; + + result.max_output_boxes_per_class = nms5->max_boxes_output_from_input(); + result.iou_threshold = nms5->iou_threshold_from_input(); + result.score_threshold = nms5->score_threshold_from_input(); + result.soft_nms_sigma = nms5->soft_nms_sigma_from_input(); + + auto selected_indices_shape = + infer_selected_indices_shape(inputs, result.max_output_boxes_per_class); + result.out_shape = selected_indices_shape.to_shape(); + + result.boxes_shape = inputs[boxes_port]->get_shape(); + result.scores_shape = inputs[scores_port]->get_shape(); + + result.boxes_data = prepare_boxes_data( + inputs[boxes_port], result.boxes_shape, nms5->get_box_encoding()); + result.scores_data = prepare_scores_data(inputs[scores_port], result.scores_shape); + + result.out_shape_size = shape_size(result.out_shape); + + result.sort_result_descending = nms5->get_sort_result_descending(); + + result.output_type = nms5->get_output_type(); + + return result; + } + + void non_max_suppression(const float* boxes_data, + const Shape& boxes_data_shape, + const float* scores_data, + const Shape& scores_data_shape, + int64_t max_output_boxes_per_class, + float iou_threshold, + float score_threshold, + float soft_nms_sigma, + int64_t* selected_indices, + const Shape& selected_indices_shape, + float* selected_scores, + const Shape& selected_scores_shape, + int64_t* valid_outputs, + const bool sort_result_descending) + { + float scale = 0.0f; + if (soft_nms_sigma > 0.0f) + { + scale = -0.5f / soft_nms_sigma; + } + + auto func = [iou_threshold, scale](float iou) { + const float weight = std::exp(scale * iou * iou); + return iou <= iou_threshold ? weight : 0.0f; + }; + + // boxes shape: {num_batches, num_boxes, 4} + // scores shape: {num_batches, num_classes, num_boxes} + int64_t num_batches = static_cast(scores_data_shape[0]); + int64_t num_classes = static_cast(scores_data_shape[1]); + int64_t num_boxes = static_cast(boxes_data_shape[1]); + + SelectedIndex* selected_indices_ptr = + reinterpret_cast(selected_indices); + SelectedScore* selected_scores_ptr = + reinterpret_cast(selected_scores); + + size_t boxes_per_class = static_cast(max_output_boxes_per_class); + + int64_t num_of_valid_boxes = 0; + + std::vector filteredBoxes; + + for (int64_t batch = 0; batch < num_batches; batch++) + { + const float* boxesPtr = boxes_data + batch * num_boxes * 4; + Rectangle* r = reinterpret_cast(const_cast(boxesPtr)); + + for (int64_t class_idx = 0; class_idx < num_classes; class_idx++) + { + const float* scoresPtr = + scores_data + batch * (num_classes * num_boxes) + class_idx * num_boxes; + + std::vector candidate_boxes; + candidate_boxes.reserve(num_boxes); + + for (size_t box_idx = 0; box_idx < num_boxes; box_idx++) + { + if (scoresPtr[box_idx] > score_threshold) + { + candidate_boxes.emplace_back( + r[box_idx], box_idx, scoresPtr[box_idx], 0, batch, class_idx); + } + } + + std::priority_queue sorted_boxes(std::less(), + std::move(candidate_boxes)); + + std::vector selected; + // Get the next box with top score, filter by iou_threshold + + BoxInfo next_candidate; + float original_score; + + while (!sorted_boxes.empty() && selected.size() < boxes_per_class) + { + next_candidate = sorted_boxes.top(); + original_score = next_candidate.score; + sorted_boxes.pop(); + + bool should_hard_suppress = false; + for (int64_t j = static_cast(selected.size()) - 1; + j >= next_candidate.suppress_begin_index; + --j) + { + float iou = + intersectionOverUnion(next_candidate.box, selected[j].box); + next_candidate.score *= func(iou); + + if (iou >= iou_threshold) + { + should_hard_suppress = true; + break; + } + + if (next_candidate.score <= score_threshold) + { + break; + } + } + + next_candidate.suppress_begin_index = selected.size(); + + if (!should_hard_suppress) + { + if (next_candidate.score == original_score) + { + selected.push_back(next_candidate); + continue; + } + if (next_candidate.score > score_threshold) + { + sorted_boxes.push(next_candidate); + } + } + } + + for (const auto& box_info : selected) + { + filteredBoxes.push_back(box_info); + } + } + } + + if (sort_result_descending) + { + std::sort(filteredBoxes.begin(), + filteredBoxes.end(), + [](const BoxInfo& l, const BoxInfo& r) { return l.score > r.score; }); + } + + size_t max_num_of_selected_indices = selected_indices_shape[0]; + size_t output_size = std::min(filteredBoxes.size(), max_num_of_selected_indices); + + *valid_outputs = output_size; + + size_t idx; + for (idx = 0; idx < output_size; idx++) + { + const auto& box_info = filteredBoxes[idx]; + SelectedIndex selected_index{ + box_info.batch_index, box_info.class_index, box_info.index}; + SelectedScore selected_score{static_cast(box_info.batch_index), + static_cast(box_info.class_index), + box_info.score}; + + selected_indices_ptr[idx] = selected_index; + selected_scores_ptr[idx] = selected_score; + } + + SelectedIndex selected_index_filler{0, 0, 0}; + SelectedScore selected_score_filler{0.0f, 0.0f, 0.0f}; + for (; idx < max_num_of_selected_indices; idx++) + { + selected_indices_ptr[idx] = selected_index_filler; + selected_scores_ptr[idx] = selected_score_filler; + } + } + + void nms5_postprocessing(const HostTensorVector& outputs, + const ngraph::element::Type output_type, + const std::vector& selected_indices, + const std::vector& selected_scores, + int64_t valid_outputs, + const ngraph::element::Type selected_scores_type) + { + outputs[0]->set_element_type(output_type); + outputs[0]->set_shape(Shape{static_cast(valid_outputs), 3}); + + size_t num_of_outputs = outputs.size(); + + if (num_of_outputs >= 2) + { + outputs[1]->set_element_type(selected_scores_type); + outputs[1]->set_shape(Shape{static_cast(valid_outputs), 3}); + } + + if (num_of_outputs >= 3) + { + outputs[2]->set_element_type(output_type); + outputs[2]->set_shape(Shape{1}); + } + + size_t selected_size = valid_outputs * 3; + + if (output_type == ngraph::element::i64) + { + int64_t* indices_ptr = outputs[0]->get_data_ptr(); + memcpy(indices_ptr, selected_indices.data(), selected_size * sizeof(int64_t)); + } + else + { + int32_t* indices_ptr = outputs[0]->get_data_ptr(); + for (size_t i = 0; i < selected_size; ++i) + { + indices_ptr[i] = static_cast(selected_indices[i]); + } + } + + if (num_of_outputs < 2) + { + return; + } + + size_t selected_scores_size = selected_scores.size(); + + switch (selected_scores_type) + { + case element::Type_t::bf16: + { + bfloat16* scores_ptr = outputs[1]->get_data_ptr(); + for (size_t i = 0; i < selected_scores_size; ++i) + { + scores_ptr[i] = bfloat16(selected_scores[i]); + } + } + break; + case element::Type_t::f16: + { + float16* scores_ptr = outputs[1]->get_data_ptr(); + for (size_t i = 0; i < selected_scores_size; ++i) + { + scores_ptr[i] = float16(selected_scores[i]); + } + } + break; + case element::Type_t::f32: + { + float* scores_ptr = outputs[1]->get_data_ptr(); + memcpy(scores_ptr, selected_scores.data(), selected_size * sizeof(float)); + } + break; + default:; + } + + if (num_of_outputs < 3) + { + return; + } + + if (output_type == ngraph::element::i64) + { + int64_t* valid_outputs_ptr = outputs[2]->get_data_ptr(); + *valid_outputs_ptr = valid_outputs; + } + else + { + int32_t* valid_outputs_ptr = outputs[2]->get_data_ptr(); + *valid_outputs_ptr = static_cast(valid_outputs); + } + } + } + } +} diff --git a/ngraph/core/src/op/non_max_suppression.cpp b/ngraph/core/src/op/non_max_suppression.cpp index d631545db20c40..f5b94ac263c70f 100644 --- a/ngraph/core/src/op/non_max_suppression.cpp +++ b/ngraph/core/src/op/non_max_suppression.cpp @@ -19,8 +19,11 @@ #include "ngraph/attribute_visitor.hpp" #include "ngraph/op/constant.hpp" #include "ngraph/op/util/op_types.hpp" +#include "ngraph/runtime/reference/non_max_suppression.hpp" +#include "ngraph/type/bfloat16.hpp" +#include "ngraph/type/float16.hpp" +#include "ngraph/util.hpp" -using namespace std; using namespace ngraph; // ------------------------------ V1 ------------------------------ @@ -58,7 +61,7 @@ op::v1::NonMaxSuppression::NonMaxSuppression( constructor_validate_and_infer_types(); } -shared_ptr +std::shared_ptr op::v1::NonMaxSuppression::clone_with_new_inputs(const OutputVector& new_args) const { check_new_args_count(this, new_args); @@ -256,7 +259,7 @@ op::v3::NonMaxSuppression::NonMaxSuppression( constructor_validate_and_infer_types(); } -shared_ptr +std::shared_ptr op::v3::NonMaxSuppression::clone_with_new_inputs(const OutputVector& new_args) const { check_new_args_count(this, new_args); @@ -475,7 +478,7 @@ op::v4::NonMaxSuppression::NonMaxSuppression( constructor_validate_and_infer_types(); } -shared_ptr +std::shared_ptr op::v4::NonMaxSuppression::clone_with_new_inputs(const OutputVector& new_args) const { check_new_args_count(this, new_args); @@ -542,12 +545,7 @@ op::v5::NonMaxSuppression::NonMaxSuppression( const op::v5::NonMaxSuppression::BoxEncodingType box_encoding, const bool sort_result_descending, const element::Type& output_type) - : Op({boxes, - scores, - op::Constant::create(element::i64, Shape{}, {0}), - op::Constant::create(element::f32, Shape{}, {.0f}), - op::Constant::create(element::f32, Shape{}, {.0f}), - op::Constant::create(element::f32, Shape{}, {.0f})}) + : Op({boxes, scores}) , m_box_encoding{box_encoding} , m_sort_result_descending{sort_result_descending} , m_output_type{output_type} @@ -562,12 +560,7 @@ op::v5::NonMaxSuppression::NonMaxSuppression( const op::v5::NonMaxSuppression::BoxEncodingType box_encoding, const bool sort_result_descending, const element::Type& output_type) - : Op({boxes, - scores, - max_output_boxes_per_class, - op::Constant::create(element::f32, Shape{}, {.0f}), - op::Constant::create(element::f32, Shape{}, {.0f}), - op::Constant::create(element::f32, Shape{}, {.0f})}) + : Op({boxes, scores, max_output_boxes_per_class}) , m_box_encoding{box_encoding} , m_sort_result_descending{sort_result_descending} , m_output_type{output_type} @@ -583,12 +576,7 @@ op::v5::NonMaxSuppression::NonMaxSuppression( const op::v5::NonMaxSuppression::BoxEncodingType box_encoding, const bool sort_result_descending, const element::Type& output_type) - : Op({boxes, - scores, - max_output_boxes_per_class, - iou_threshold, - op::Constant::create(element::f32, Shape{}, {.0f}), - op::Constant::create(element::f32, Shape{}, {.0f})}) + : Op({boxes, scores, max_output_boxes_per_class, iou_threshold}) , m_box_encoding{box_encoding} , m_sort_result_descending{sort_result_descending} , m_output_type{output_type} @@ -605,12 +593,7 @@ op::v5::NonMaxSuppression::NonMaxSuppression( const op::v5::NonMaxSuppression::BoxEncodingType box_encoding, const bool sort_result_descending, const element::Type& output_type) - : Op({boxes, - scores, - max_output_boxes_per_class, - iou_threshold, - score_threshold, - op::Constant::create(element::f32, Shape{}, {.0f})}) + : Op({boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold}) , m_box_encoding{box_encoding} , m_sort_result_descending{sort_result_descending} , m_output_type{output_type} @@ -641,7 +624,7 @@ op::v5::NonMaxSuppression::NonMaxSuppression( constructor_validate_and_infer_types(); } -shared_ptr +std::shared_ptr op::v5::NonMaxSuppression::clone_with_new_inputs(const OutputVector& new_args) const { check_new_args_count(this, new_args); @@ -649,28 +632,81 @@ shared_ptr new_args.size() >= 2 && new_args.size() <= 6, "Number of inputs must be 2, 3, 4, 5 or 6"); - const auto& arg2 = new_args.size() > 2 - ? new_args.at(2) - : ngraph::op::Constant::create(element::i64, Shape{}, {0}); - const auto& arg3 = new_args.size() > 3 - ? new_args.at(3) - : ngraph::op::Constant::create(element::f32, Shape{}, {.0f}); - const auto& arg4 = new_args.size() > 4 - ? new_args.at(4) - : ngraph::op::Constant::create(element::f32, Shape{}, {.0f}); - const auto& arg5 = new_args.size() > 5 - ? new_args.at(5) - : ngraph::op::Constant::create(element::f32, Shape{}, {.0f}); + switch (new_args.size()) + { + case 2: + return std::make_shared(new_args.at(0), + new_args.at(1), + m_box_encoding, + m_sort_result_descending, + m_output_type); + break; + case 3: + return std::make_shared(new_args.at(0), + new_args.at(1), + new_args.at(2), + m_box_encoding, + m_sort_result_descending, + m_output_type); + break; + case 4: + return std::make_shared(new_args.at(0), + new_args.at(1), + new_args.at(2), + new_args.at(3), + m_box_encoding, + m_sort_result_descending, + m_output_type); + break; + case 5: + return std::make_shared(new_args.at(0), + new_args.at(1), + new_args.at(2), + new_args.at(3), + new_args.at(4), + m_box_encoding, + m_sort_result_descending, + m_output_type); + break; + default: + return std::make_shared(new_args.at(0), + new_args.at(1), + new_args.at(2), + new_args.at(3), + new_args.at(4), + new_args.at(5), + m_box_encoding, + m_sort_result_descending, + m_output_type); + break; + } +} - return std::make_shared(new_args.at(0), - new_args.at(1), - arg2, - arg3, - arg4, - arg5, - m_box_encoding, - m_sort_result_descending, - m_output_type); +namespace +{ + constexpr size_t boxes_port = 0; + constexpr size_t scores_port = 1; + constexpr size_t max_output_boxes_port = 2; + constexpr size_t iou_threshold_port = 3; + constexpr size_t score_threshold_port = 4; + constexpr size_t soft_nms_sigma_port = 5; + + inline bool is_float_type_admissible(const element::Type& t) + { + return t == element::f32 || t == element::f16 || t == element::bf16; + } + + inline bool is_scalar_or_1d_tensor_with_1_element(const PartialShape& p) + { + if (p.is_dynamic()) + { + return false; + } + + Shape shape = p.to_shape(); + + return is_scalar(shape) || (is_vector(shape) && (shape[0] == 1)); + } } void op::v5::NonMaxSuppression::validate() @@ -687,6 +723,14 @@ void op::v5::NonMaxSuppression::validate() return; } + NODE_VALIDATION_CHECK(this, + is_float_type_admissible(get_input_element_type(0)), + "Expected bf16, fp16 or fp32 as element type for the 'boxes' input."); + + NODE_VALIDATION_CHECK(this, + is_float_type_admissible(get_input_element_type(1)), + "Expected bf16, fp16 or fp32 as element type for the 'scores' input."); + NODE_VALIDATION_CHECK(this, boxes_ps.rank().is_static() && boxes_ps.rank().get_length() == 3, "Expected a 3D tensor for the 'boxes' input. Got: ", @@ -700,29 +744,39 @@ void op::v5::NonMaxSuppression::validate() if (inputs().size() >= 3) { const auto max_boxes_ps = get_input_partial_shape(2); - NODE_VALIDATION_CHECK(this, - max_boxes_ps.is_dynamic() || is_scalar(max_boxes_ps.to_shape()), - "Expected a scalar for the 'max_output_boxes_per_class' input. Got: ", - max_boxes_ps); + NODE_VALIDATION_CHECK( + this, + max_boxes_ps.is_dynamic() || is_scalar_or_1d_tensor_with_1_element(max_boxes_ps), + "Expected 0D or 1D tensor for the 'max_output_boxes_per_class' input. " + "Got: ", + max_boxes_ps); } if (inputs().size() >= 4) { const auto iou_threshold_ps = get_input_partial_shape(3); + NODE_VALIDATION_CHECK(this, + is_float_type_admissible(get_input_element_type(3)), + "Expected bf16, fp16 or fp32 as element type for the " + "'iou_threshold' input."); NODE_VALIDATION_CHECK(this, iou_threshold_ps.is_dynamic() || - is_scalar(iou_threshold_ps.to_shape()), - "Expected a scalar for the 'iou_threshold' input. Got: ", + is_scalar_or_1d_tensor_with_1_element(iou_threshold_ps), + "Expected 0D or 1D tensor for the 'iou_threshold' input. Got: ", iou_threshold_ps); } if (inputs().size() >= 5) { const auto score_threshold_ps = get_input_partial_shape(4); + NODE_VALIDATION_CHECK(this, + is_float_type_admissible(get_input_element_type(4)), + "Expected bf16, fp16 or fp32 as element type for the " + "'score_threshold_ps' input."); NODE_VALIDATION_CHECK(this, score_threshold_ps.is_dynamic() || - is_scalar(score_threshold_ps.to_shape()), - "Expected a scalar for the 'score_threshold' input. Got: ", + is_scalar_or_1d_tensor_with_1_element(score_threshold_ps), + "Expected 0D or 1D tensor for the 'score_threshold' input. Got: ", score_threshold_ps); } @@ -730,8 +784,13 @@ void op::v5::NonMaxSuppression::validate() { const auto soft_nms_sigma = get_input_partial_shape(5); NODE_VALIDATION_CHECK(this, - soft_nms_sigma.is_dynamic() || is_scalar(soft_nms_sigma.to_shape()), - "Expected a scalar for the 'soft_nms_sigma' input. Got: ", + is_float_type_admissible(get_input_element_type(5)), + "Expected bf16, fp16 or fp32 as element type for the " + "'soft_nms_sigma' input."); + NODE_VALIDATION_CHECK(this, + soft_nms_sigma.is_dynamic() || + is_scalar_or_1d_tensor_with_1_element(soft_nms_sigma), + "Expected 0D or 1D tensor for the 'soft_nms_sigma' input. Got: ", soft_nms_sigma); } @@ -764,23 +823,27 @@ int64_t op::v5::NonMaxSuppression::max_boxes_output_from_input() const { int64_t max_output_boxes{0}; + if (inputs().size() < 3) + { + return 0; + } + const auto max_output_boxes_input = - as_type_ptr(input_value(2).get_node_shared_ptr()); + as_type_ptr(input_value(max_output_boxes_port).get_node_shared_ptr()); max_output_boxes = max_output_boxes_input->cast_vector().at(0); return max_output_boxes; } -static constexpr size_t boxes_port = 0; -static constexpr size_t scores_port = 1; -static constexpr size_t iou_threshold_port = 3; -static constexpr size_t score_threshold_port = 4; -static constexpr size_t soft_nms_sigma_port = 5; - float op::v5::NonMaxSuppression::iou_threshold_from_input() const { float iou_threshold = 0.0f; + if (inputs().size() < 4) + { + return iou_threshold; + } + const auto iou_threshold_input = as_type_ptr(input_value(iou_threshold_port).get_node_shared_ptr()); iou_threshold = iou_threshold_input->cast_vector().at(0); @@ -792,6 +855,11 @@ float op::v5::NonMaxSuppression::score_threshold_from_input() const { float score_threshold = 0.0f; + if (inputs().size() < 5) + { + return score_threshold; + } + const auto score_threshold_input = as_type_ptr(input_value(score_threshold_port).get_node_shared_ptr()); score_threshold = score_threshold_input->cast_vector().at(0); @@ -803,6 +871,11 @@ float op::v5::NonMaxSuppression::soft_nms_sigma_from_input() const { float soft_nms_sigma = 0.0f; + if (inputs().size() < 6) + { + return soft_nms_sigma; + } + const auto soft_nms_sigma_input = as_type_ptr(input_value(soft_nms_sigma_port).get_node_shared_ptr()); soft_nms_sigma = soft_nms_sigma_input->cast_vector().at(0); diff --git a/ngraph/python/src/ngraph/opset5/__init__.py b/ngraph/python/src/ngraph/opset5/__init__.py index 2b8fac3ea3f9e5..b9035b4114f4a6 100644 --- a/ngraph/python/src/ngraph/opset5/__init__.py +++ b/ngraph/python/src/ngraph/opset5/__init__.py @@ -98,7 +98,7 @@ from ngraph.opset1.ops import multiply from ngraph.opset2.ops import mvn from ngraph.opset1.ops import negative -from ngraph.opset4.ops import non_max_suppression +from ngraph.opset5.ops import non_max_suppression from ngraph.opset3.ops import non_zero from ngraph.opset1.ops import normalize_l2 from ngraph.opset1.ops import not_equal diff --git a/ngraph/python/src/ngraph/opset5/ops.py b/ngraph/python/src/ngraph/opset5/ops.py index f6cf2c0745c81a..56fee6bb249348 100644 --- a/ngraph/python/src/ngraph/opset5/ops.py +++ b/ngraph/python/src/ngraph/opset5/ops.py @@ -118,6 +118,58 @@ def log_softmax(data: NodeInput, axis: int, name: Optional[str] = None) -> Node: return _get_node_factory_opset5().create("LogSoftmax", [as_node(data)], {"axis": axis}) +@nameable_op +def non_max_suppression( + boxes: NodeInput, + scores: NodeInput, + max_output_boxes_per_class: Optional[NodeInput] = None, + iou_threshold: Optional[NodeInput] = None, + score_threshold: Optional[NodeInput] = None, + soft_nms_sigma: Optional[NodeInput] = None, + box_encoding: str = "corner", + sort_result_descending: bool = True, + output_type: str = "i64", + name: Optional[str] = None, +) -> Node: + """Return a node which performs NonMaxSuppression. + + :param boxes: Tensor with box coordinates. + :param scores: Tensor with box scores. + :param max_output_boxes_per_class: Tensor Specifying maximum number of boxes + to be selected per class. + :param iou_threshold: Tensor specifying intersection over union threshold + :param score_threshold: Tensor specifying minimum score to consider box for the processing. + :param soft_nms_sigma: Tensor specifying the sigma parameter for Soft-NMS. + :param box_encoding: Format of boxes data encoding. + :param sort_result_descending: Flag that specifies whenever it is necessary to sort selected + boxes across batches or not. + :param output_type: Output element type. + :return: The new node which performs NonMaxSuppression + """ + if max_output_boxes_per_class is None: + max_output_boxes_per_class = make_constant_node(0, np.int64) + if iou_threshold is None: + iou_threshold = make_constant_node(0, np.float32) + if score_threshold is None: + score_threshold = make_constant_node(0, np.float32) + if soft_nms_sigma is None: + inputs = as_nodes( + boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold + ) + else: + inputs = as_nodes( + boxes, scores, max_output_boxes_per_class, iou_threshold, score_threshold, soft_nms_sigma + ) + + attributes = { + "box_encoding": box_encoding, + "sort_result_descending": sort_result_descending, + "output_type": output_type, + } + + return _get_node_factory_opset5().create("NonMaxSuppression", inputs, attributes) + + @nameable_op def round(data: NodeInput, mode: str = "half_to_even", name: Optional[str] = None) -> Node: """Apply Round operation on each element of input tensor. diff --git a/ngraph/python/tests/test_ngraph/test_reduction.py b/ngraph/python/tests/test_ngraph/test_reduction.py index b4bac7f02798f0..86384599fd1eca 100644 --- a/ngraph/python/tests/test_ngraph/test_reduction.py +++ b/ngraph/python/tests/test_ngraph/test_reduction.py @@ -15,6 +15,7 @@ # ****************************************************************************** import numpy as np import pytest +from _pyngraph import PartialShape import ngraph as ng from tests.runtime import get_runtime @@ -104,15 +105,16 @@ def test_non_max_suppression(): boxes_shape = [1, 1000, 4] scores_shape = [1, 1, 1000] - expected_shape = [0, 3] boxes_parameter = ng.parameter(boxes_shape, name="Boxes", dtype=np.float32) scores_parameter = ng.parameter(scores_shape, name="Scores", dtype=np.float32) node = ng.non_max_suppression(boxes_parameter, scores_parameter) assert node.get_type_name() == "NonMaxSuppression" - assert node.get_output_size() == 1 - assert list(node.get_output_shape(0)) == expected_shape + assert node.get_output_size() == 3 + assert node.get_output_partial_shape(0).same_scheme(PartialShape([-1, 3])) + assert node.get_output_partial_shape(1).same_scheme(PartialShape([-1, 3])) + assert list(node.get_output_shape(2)) == [1] def test_non_zero(): diff --git a/ngraph/test/CMakeLists.txt b/ngraph/test/CMakeLists.txt index 8e486b6e865f77..d7760dc190d922 100644 --- a/ngraph/test/CMakeLists.txt +++ b/ngraph/test/CMakeLists.txt @@ -308,6 +308,7 @@ set(MULTI_TEST_SRC backend/negative.in.cpp backend/node_name.in.cpp backend/normalize_l2.in.cpp + backend/non_max_suppression.in.cpp backend/non_zero.in.cpp backend/numeric.in.cpp backend/one_hot.in.cpp diff --git a/ngraph/test/backend/non_max_suppression.in.cpp b/ngraph/test/backend/non_max_suppression.in.cpp new file mode 100644 index 00000000000000..e258d272e41dcd --- /dev/null +++ b/ngraph/test/backend/non_max_suppression.in.cpp @@ -0,0 +1,614 @@ +//***************************************************************************** +// Copyright 2020 Intel Corporation +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +//***************************************************************************** + +// clang-format off +#ifdef ${BACKEND_NAME}_FLOAT_TOLERANCE_BITS +#define DEFAULT_FLOAT_TOLERANCE_BITS ${BACKEND_NAME}_FLOAT_TOLERANCE_BITS +#endif + +#ifdef ${BACKEND_NAME}_DOUBLE_TOLERANCE_BITS +#define DEFAULT_DOUBLE_TOLERANCE_BITS ${BACKEND_NAME}_DOUBLE_TOLERANCE_BITS +#endif +// clang-format on + +#include "gtest/gtest.h" +#include "runtime/backend.hpp" +#include "ngraph/runtime/tensor.hpp" +#include "ngraph/ngraph.hpp" +#include "util/all_close.hpp" +#include "util/all_close_f.hpp" +#include "util/known_element_types.hpp" +#include "util/ndarray.hpp" +#include "util/test_control.hpp" +#include "util/test_tools.hpp" + +NGRAPH_SUPPRESS_DEPRECATED_START + +using namespace std; +using namespace ngraph; + +static string s_manifest = "${MANIFEST}"; + +NGRAPH_TEST(${BACKEND_NAME}, nonmaxsuppression_center_point_box_format) +{ + std::vector boxes_data = {0.5, 0.5, 1.0, 1.0, 0.5, 0.6, 1.0, 1.0, + 0.5, 0.4, 1.0, 1.0, 0.5, 10.5, 1.0, 1.0, + 0.5, 10.6, 1.0, 1.0, 0.5, 100.5, 1.0, 1.0}; + + std::vector scores_data = {0.9, 0.75, 0.6, 0.95, 0.5, 0.3}; + + const int64_t max_output_boxes_per_class_data = 3; + const float iou_threshold_data = 0.5f; + const float score_threshold_data = 0.0f; + const auto box_encoding = op::v5::NonMaxSuppression::BoxEncodingType::CENTER; + const auto boxes_shape = Shape{1, 6, 4}; + const auto scores_shape = Shape{1, 1, 6}; + + const auto boxes = make_shared(element::f32, boxes_shape); + const auto scores = make_shared(element::f32, scores_shape); + auto max_output_boxes_per_class = + op::Constant::create(element::i64, Shape{}, {max_output_boxes_per_class_data}); + auto iou_threshold = op::Constant::create(element::f32, Shape{}, {iou_threshold_data}); + auto score_threshold = + op::Constant::create(element::f32, Shape{}, {score_threshold_data}); + auto soft_nms_sigma = op::Constant::create(element::f32, Shape{}, {0.0f}); + auto nms = make_shared(boxes, + scores, + max_output_boxes_per_class, + iou_threshold, + score_threshold, + soft_nms_sigma, + box_encoding, + false); + + auto f = make_shared(nms, ParameterVector{boxes, scores}); + + auto backend = runtime::Backend::create("${BACKEND_NAME}"); + + auto selected_indeces = backend->create_tensor(element::i64, Shape{3, 3}); + auto selected_scores = backend->create_tensor(element::f32, Shape{3, 3}); + auto valid_outputs = backend->create_tensor(element::i64, Shape{1}); + + auto backend_boxes = backend->create_tensor(element::f32, boxes_shape); + auto backend_scores = backend->create_tensor(element::f32, scores_shape); + copy_data(backend_boxes, boxes_data); + copy_data(backend_scores, scores_data); + + auto handle = backend->compile(f); + + handle->call({selected_indeces, selected_scores, valid_outputs}, + {backend_boxes, backend_scores}); + + auto selected_indeces_value = read_vector(selected_indeces); + auto selected_scores_value = read_vector(selected_scores); + auto valid_outputs_value = read_vector(valid_outputs); + + std::vector expected_selected_indices = {0, 0, 3, 0, 0, 0, 0, 0, 5}; + std::vector expected_selected_scores = {0.0, 0.0, 0.95, 0.0, 0.0, 0.9, 0.0, 0.0, 0.3}; + std::vector expected_valid_outputs = {3}; + + EXPECT_EQ(expected_selected_indices, selected_indeces_value); + EXPECT_EQ(expected_selected_scores, selected_scores_value); + EXPECT_EQ(expected_valid_outputs, valid_outputs_value); +} + +NGRAPH_TEST(${BACKEND_NAME}, nonmaxsuppression_flipped_coordinates) +{ + std::vector boxes_data = {1.0, 1.0, 0.0, 0.0, 0.0, 0.1, 1.0, 1.1, + 0.0, 0.9, 1.0, -0.1, 0.0, 10.0, 1.0, 11.0, + 1.0, 10.1, 0.0, 11.1, 1.0, 101.0, 0.0, 100.0}; + + std::vector scores_data = {0.9, 0.75, 0.6, 0.95, 0.5, 0.3}; + + const int64_t max_output_boxes_per_class_data = 3; + const float iou_threshold_data = 0.5f; + const float score_threshold_data = 0.0f; + const auto box_encoding = op::v5::NonMaxSuppression::BoxEncodingType::CORNER; + const auto boxes_shape = Shape{1, 6, 4}; + const auto scores_shape = Shape{1, 1, 6}; + + const auto boxes = make_shared(element::f32, boxes_shape); + const auto scores = make_shared(element::f32, scores_shape); + auto max_output_boxes_per_class = + op::Constant::create(element::i64, Shape{}, {max_output_boxes_per_class_data}); + auto iou_threshold = op::Constant::create(element::f32, Shape{}, {iou_threshold_data}); + auto score_threshold = + op::Constant::create(element::f32, Shape{}, {score_threshold_data}); + auto soft_nms_sigma = op::Constant::create(element::f32, Shape{}, {0.0f}); + auto nms = make_shared(boxes, + scores, + max_output_boxes_per_class, + iou_threshold, + score_threshold, + soft_nms_sigma, + box_encoding, + false); + + auto f = make_shared(nms, ParameterVector{boxes, scores}); + + auto backend = runtime::Backend::create("${BACKEND_NAME}"); + + auto selected_indeces = backend->create_tensor(element::i64, Shape{3, 3}); + auto selected_scores = backend->create_tensor(element::f32, Shape{3, 3}); + auto valid_outputs = backend->create_tensor(element::i64, Shape{1}); + + auto backend_boxes = backend->create_tensor(element::f32, boxes_shape); + auto backend_scores = backend->create_tensor(element::f32, scores_shape); + copy_data(backend_boxes, boxes_data); + copy_data(backend_scores, scores_data); + + auto handle = backend->compile(f); + + handle->call({selected_indeces, selected_scores, valid_outputs}, + {backend_boxes, backend_scores}); + + auto selected_indeces_value = read_vector(selected_indeces); + auto selected_scores_value = read_vector(selected_scores); + auto valid_outputs_value = read_vector(valid_outputs); + + std::vector expected_selected_indices = {0, 0, 3, 0, 0, 0, 0, 0, 5}; + std::vector expected_selected_scores = {0.0, 0.0, 0.95, 0.0, 0.0, 0.9, 0.0, 0.0, 0.3}; + std::vector expected_valid_outputs = {3}; + + EXPECT_EQ(expected_selected_indices, selected_indeces_value); + EXPECT_EQ(expected_selected_scores, selected_scores_value); + EXPECT_EQ(expected_valid_outputs, valid_outputs_value); +} + +NGRAPH_TEST(${BACKEND_NAME}, nonmaxsuppression_identical_boxes) +{ + std::vector boxes_data = {0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, + 1.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 1.0, + 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, + 1.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 1.0}; + + std::vector scores_data = {0.9, 0.9, 0.9, 0.9, 0.9, 0.9, 0.9, 0.9, 0.9, 0.9}; + + const int64_t max_output_boxes_per_class_data = 3; + const float iou_threshold_data = 0.5f; + const float score_threshold_data = 0.0f; + const auto box_encoding = op::v5::NonMaxSuppression::BoxEncodingType::CORNER; + const auto boxes_shape = Shape{1, 10, 4}; + const auto scores_shape = Shape{1, 1, 10}; + + const auto boxes = make_shared(element::f32, boxes_shape); + const auto scores = make_shared(element::f32, scores_shape); + auto max_output_boxes_per_class = + op::Constant::create(element::i64, Shape{}, {max_output_boxes_per_class_data}); + auto iou_threshold = op::Constant::create(element::f32, Shape{}, {iou_threshold_data}); + auto score_threshold = + op::Constant::create(element::f32, Shape{}, {score_threshold_data}); + auto soft_nms_sigma = op::Constant::create(element::f32, Shape{}, {0.0f}); + auto nms = make_shared(boxes, + scores, + max_output_boxes_per_class, + iou_threshold, + score_threshold, + soft_nms_sigma, + box_encoding, + false); + + auto f = make_shared(nms, ParameterVector{boxes, scores}); + + auto backend = runtime::Backend::create("${BACKEND_NAME}"); + + auto selected_indeces = backend->create_tensor(element::i64, Shape{1, 3}); + auto selected_scores = backend->create_tensor(element::f32, Shape{1, 3}); + auto valid_outputs = backend->create_tensor(element::i64, Shape{1}); + + auto backend_boxes = backend->create_tensor(element::f32, boxes_shape); + auto backend_scores = backend->create_tensor(element::f32, scores_shape); + copy_data(backend_boxes, boxes_data); + copy_data(backend_scores, scores_data); + + auto handle = backend->compile(f); + + handle->call({selected_indeces, selected_scores, valid_outputs}, + {backend_boxes, backend_scores}); + + auto selected_indeces_value = read_vector(selected_indeces); + auto selected_scores_value = read_vector(selected_scores); + auto valid_outputs_value = read_vector(valid_outputs); + + std::vector expected_selected_indices = {0, 0, 0}; + std::vector expected_selected_scores = {0.0, 0.0, 0.9}; + std::vector expected_valid_outputs = {1}; + + EXPECT_EQ(expected_selected_indices, selected_indeces_value); + EXPECT_EQ(expected_selected_scores, selected_scores_value); + EXPECT_EQ(expected_valid_outputs, valid_outputs_value); +} + +NGRAPH_TEST(${BACKEND_NAME}, nonmaxsuppression_limit_output_size) +{ + std::vector boxes_data = {0.0, 0.0, 1.0, 1.0, 0.0, 0.1, 1.0, 1.1, + 0.0, -0.1, 1.0, 0.9, 0.0, 10.0, 1.0, 11.0, + 0.0, 10.1, 1.0, 11.1, 0.0, 100.0, 1.0, 101.0}; + + std::vector scores_data = {0.9, 0.75, 0.6, 0.95, 0.5, 0.3}; + + const int64_t max_output_boxes_per_class_data = 2; + const float iou_threshold_data = 0.5f; + const float score_threshold_data = 0.0f; + const auto box_encoding = op::v5::NonMaxSuppression::BoxEncodingType::CORNER; + const auto boxes_shape = Shape{1, 6, 4}; + const auto scores_shape = Shape{1, 1, 6}; + + const auto boxes = make_shared(element::f32, boxes_shape); + const auto scores = make_shared(element::f32, scores_shape); + auto max_output_boxes_per_class = + op::Constant::create(element::i64, Shape{}, {max_output_boxes_per_class_data}); + auto iou_threshold = op::Constant::create(element::f32, Shape{}, {iou_threshold_data}); + auto score_threshold = + op::Constant::create(element::f32, Shape{}, {score_threshold_data}); + auto soft_nms_sigma = op::Constant::create(element::f32, Shape{}, {0.0f}); + auto nms = make_shared(boxes, + scores, + max_output_boxes_per_class, + iou_threshold, + score_threshold, + soft_nms_sigma, + box_encoding, + false); + + auto f = make_shared(nms, ParameterVector{boxes, scores}); + + auto backend = runtime::Backend::create("${BACKEND_NAME}"); + + auto selected_indeces = backend->create_tensor(element::i64, Shape{2, 3}); + auto selected_scores = backend->create_tensor(element::f32, Shape{2, 3}); + auto valid_outputs = backend->create_tensor(element::i64, Shape{1}); + + auto backend_boxes = backend->create_tensor(element::f32, boxes_shape); + auto backend_scores = backend->create_tensor(element::f32, scores_shape); + copy_data(backend_boxes, boxes_data); + copy_data(backend_scores, scores_data); + + auto handle = backend->compile(f); + + handle->call({selected_indeces, selected_scores, valid_outputs}, + {backend_boxes, backend_scores}); + + auto selected_indeces_value = read_vector(selected_indeces); + auto selected_scores_value = read_vector(selected_scores); + auto valid_outputs_value = read_vector(valid_outputs); + + std::vector expected_selected_indices = {0, 0, 3, 0, 0, 0}; + std::vector expected_selected_scores = {0.0, 0.0, 0.95, 0.0, 0.0, 0.9}; + std::vector expected_valid_outputs = {2}; + + EXPECT_EQ(expected_selected_indices, selected_indeces_value); + EXPECT_EQ(expected_selected_scores, selected_scores_value); + EXPECT_EQ(expected_valid_outputs, valid_outputs_value); +} + +NGRAPH_TEST(${BACKEND_NAME}, nonmaxsuppression_single_box) +{ + std::vector boxes_data = {0.0, 0.0, 1.0, 1.0}; + + std::vector scores_data = {0.9}; + + const int64_t max_output_boxes_per_class_data = 3; + const float iou_threshold_data = 0.5f; + const float score_threshold_data = 0.0f; + const auto box_encoding = op::v5::NonMaxSuppression::BoxEncodingType::CORNER; + const auto boxes_shape = Shape{1, 1, 4}; + const auto scores_shape = Shape{1, 1, 1}; + + const auto boxes = make_shared(element::f32, boxes_shape); + const auto scores = make_shared(element::f32, scores_shape); + auto max_output_boxes_per_class = + op::Constant::create(element::i64, Shape{}, {max_output_boxes_per_class_data}); + auto iou_threshold = op::Constant::create(element::f32, Shape{}, {iou_threshold_data}); + auto score_threshold = + op::Constant::create(element::f32, Shape{}, {score_threshold_data}); + auto soft_nms_sigma = op::Constant::create(element::f32, Shape{}, {0.0f}); + auto nms = make_shared(boxes, + scores, + max_output_boxes_per_class, + iou_threshold, + score_threshold, + soft_nms_sigma, + box_encoding, + false); + + auto f = make_shared(nms, ParameterVector{boxes, scores}); + + auto backend = runtime::Backend::create("${BACKEND_NAME}"); + + auto selected_indeces = backend->create_tensor(element::i64, Shape{1, 3}); + auto selected_scores = backend->create_tensor(element::f32, Shape{1, 3}); + auto valid_outputs = backend->create_tensor(element::i64, Shape{1}); + + auto backend_boxes = backend->create_tensor(element::f32, boxes_shape); + auto backend_scores = backend->create_tensor(element::f32, scores_shape); + copy_data(backend_boxes, boxes_data); + copy_data(backend_scores, scores_data); + + auto handle = backend->compile(f); + + handle->call({selected_indeces, selected_scores, valid_outputs}, + {backend_boxes, backend_scores}); + + auto selected_indeces_value = read_vector(selected_indeces); + auto selected_scores_value = read_vector(selected_scores); + auto valid_outputs_value = read_vector(valid_outputs); + + std::vector expected_selected_indices = {0, 0, 0}; + std::vector expected_selected_scores = {0.0, 0.0, 0.9}; + std::vector expected_valid_outputs = {1}; + + EXPECT_EQ(expected_selected_indices, selected_indeces_value); + EXPECT_EQ(expected_selected_scores, selected_scores_value); + EXPECT_EQ(expected_valid_outputs, valid_outputs_value); +} + +NGRAPH_TEST(${BACKEND_NAME}, nonmaxsuppression_suppress_by_IOU) +{ + std::vector boxes_data = {0.0, 0.0, 1.0, 1.0, 0.0, 0.1, 1.0, 1.1, + 0.0, -0.1, 1.0, 0.9, 0.0, 10.0, 1.0, 11.0, + 0.0, 10.1, 1.0, 11.1, 0.0, 100.0, 1.0, 101.0}; + + std::vector scores_data = {0.9, 0.75, 0.6, 0.95, 0.5, 0.3}; + + const int64_t max_output_boxes_per_class_data = 3; + const float iou_threshold_data = 0.5f; + const float score_threshold_data = 0.0f; + const auto box_encoding = op::v5::NonMaxSuppression::BoxEncodingType::CORNER; + const auto boxes_shape = Shape{1, 6, 4}; + const auto scores_shape = Shape{1, 1, 6}; + + const auto boxes = make_shared(element::f32, boxes_shape); + const auto scores = make_shared(element::f32, scores_shape); + auto max_output_boxes_per_class = + op::Constant::create(element::i64, Shape{}, {max_output_boxes_per_class_data}); + auto iou_threshold = op::Constant::create(element::f32, Shape{}, {iou_threshold_data}); + auto score_threshold = + op::Constant::create(element::f32, Shape{}, {score_threshold_data}); + auto soft_nms_sigma = op::Constant::create(element::f32, Shape{}, {0.0f}); + auto nms = make_shared(boxes, + scores, + max_output_boxes_per_class, + iou_threshold, + score_threshold, + soft_nms_sigma, + box_encoding, + false); + + auto f = make_shared(nms, ParameterVector{boxes, scores}); + + auto backend = runtime::Backend::create("${BACKEND_NAME}"); + + auto selected_indeces = backend->create_tensor(element::i64, Shape{3, 3}); + auto selected_scores = backend->create_tensor(element::f32, Shape{3, 3}); + auto valid_outputs = backend->create_tensor(element::i64, Shape{1}); + + auto backend_boxes = backend->create_tensor(element::f32, boxes_shape); + auto backend_scores = backend->create_tensor(element::f32, scores_shape); + copy_data(backend_boxes, boxes_data); + copy_data(backend_scores, scores_data); + + auto handle = backend->compile(f); + + handle->call({selected_indeces, selected_scores, valid_outputs}, + {backend_boxes, backend_scores}); + + auto selected_indeces_value = read_vector(selected_indeces); + auto selected_scores_value = read_vector(selected_scores); + auto valid_outputs_value = read_vector(valid_outputs); + + std::vector expected_selected_indices = {0, 0, 3, 0, 0, 0, 0, 0, 5}; + std::vector expected_selected_scores = {0.0, 0.0, 0.95, 0.0, 0.0, 0.9, 0.0, 0.0, 0.3}; + std::vector expected_valid_outputs = {3}; + + EXPECT_EQ(expected_selected_indices, selected_indeces_value); + EXPECT_EQ(expected_selected_scores, selected_scores_value); + EXPECT_EQ(expected_valid_outputs, valid_outputs_value); +} + +NGRAPH_TEST(${BACKEND_NAME}, nonmaxsuppression_suppress_by_IOU_and_scores) +{ + std::vector boxes_data = {0.0, 0.0, 1.0, 1.0, 0.0, 0.1, 1.0, 1.1, + 0.0, -0.1, 1.0, 0.9, 0.0, 10.0, 1.0, 11.0, + 0.0, 10.1, 1.0, 11.1, 0.0, 100.0, 1.0, 101.0}; + + std::vector scores_data = {0.9, 0.75, 0.6, 0.95, 0.5, 0.3}; + + const int64_t max_output_boxes_per_class_data = 3; + const float iou_threshold_data = 0.5f; + const float score_threshold_data = 0.4f; + const auto box_encoding = op::v5::NonMaxSuppression::BoxEncodingType::CORNER; + const auto boxes_shape = Shape{1, 6, 4}; + const auto scores_shape = Shape{1, 1, 6}; + + const auto boxes = make_shared(element::f32, boxes_shape); + const auto scores = make_shared(element::f32, scores_shape); + auto max_output_boxes_per_class = + op::Constant::create(element::i64, Shape{}, {max_output_boxes_per_class_data}); + auto iou_threshold = op::Constant::create(element::f32, Shape{}, {iou_threshold_data}); + auto score_threshold = + op::Constant::create(element::f32, Shape{}, {score_threshold_data}); + auto soft_nms_sigma = op::Constant::create(element::f32, Shape{}, {0.0f}); + auto nms = make_shared(boxes, + scores, + max_output_boxes_per_class, + iou_threshold, + score_threshold, + soft_nms_sigma, + box_encoding, + false); + + auto f = make_shared(nms, ParameterVector{boxes, scores}); + + auto backend = runtime::Backend::create("${BACKEND_NAME}"); + + auto selected_indeces = backend->create_tensor(element::i64, Shape{2, 3}); + auto selected_scores = backend->create_tensor(element::f32, Shape{2, 3}); + auto valid_outputs = backend->create_tensor(element::i64, Shape{1}); + + auto backend_boxes = backend->create_tensor(element::f32, boxes_shape); + auto backend_scores = backend->create_tensor(element::f32, scores_shape); + copy_data(backend_boxes, boxes_data); + copy_data(backend_scores, scores_data); + + auto handle = backend->compile(f); + + handle->call({selected_indeces, selected_scores, valid_outputs}, + {backend_boxes, backend_scores}); + + auto selected_indeces_value = read_vector(selected_indeces); + auto selected_scores_value = read_vector(selected_scores); + auto valid_outputs_value = read_vector(valid_outputs); + + std::vector expected_selected_indices = {0, 0, 3, 0, 0, 0}; + std::vector expected_selected_scores = {0.0, 0.0, 0.95, 0.0, 0.0, 0.9}; + std::vector expected_valid_outputs = {2}; + + EXPECT_EQ(expected_selected_indices, selected_indeces_value); + EXPECT_EQ(expected_selected_scores, selected_scores_value); + EXPECT_EQ(expected_valid_outputs, valid_outputs_value); +} + +NGRAPH_TEST(${BACKEND_NAME}, nonmaxsuppression_two_batches) +{ + std::vector boxes_data = { + 0.0, 0.0, 1.0, 1.0, 0.0, 0.1, 1.0, 1.1, 0.0, -0.1, 1.0, 0.9, 0.0, 10.0, 1.0, 11.0, + 0.0, 10.1, 1.0, 11.1, 0.0, 100.0, 1.0, 101.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.1, 1.0, 1.1, + 0.0, -0.1, 1.0, 0.9, 0.0, 10.0, 1.0, 11.0, 0.0, 10.1, 1.0, 11.1, 0.0, 100.0, 1.0, 101.0}; + + std::vector scores_data = { + 0.9, 0.75, 0.6, 0.95, 0.5, 0.3, 0.9, 0.75, 0.6, 0.95, 0.5, 0.3}; + + const int64_t max_output_boxes_per_class_data = 2; + const float iou_threshold_data = 0.5f; + const float score_threshold_data = 0.0f; + const auto box_encoding = op::v5::NonMaxSuppression::BoxEncodingType::CORNER; + const auto boxes_shape = Shape{2, 6, 4}; + const auto scores_shape = Shape{2, 1, 6}; + + const auto boxes = make_shared(element::f32, boxes_shape); + const auto scores = make_shared(element::f32, scores_shape); + auto max_output_boxes_per_class = + op::Constant::create(element::i64, Shape{}, {max_output_boxes_per_class_data}); + auto iou_threshold = op::Constant::create(element::f32, Shape{}, {iou_threshold_data}); + auto score_threshold = + op::Constant::create(element::f32, Shape{}, {score_threshold_data}); + auto soft_nms_sigma = op::Constant::create(element::f32, Shape{}, {0.0f}); + auto nms = make_shared(boxes, + scores, + max_output_boxes_per_class, + iou_threshold, + score_threshold, + soft_nms_sigma, + box_encoding, + false); + + auto f = make_shared(nms, ParameterVector{boxes, scores}); + + auto backend = runtime::Backend::create("${BACKEND_NAME}"); + + auto selected_indeces = backend->create_tensor(element::i64, Shape{4, 3}); + auto selected_scores = backend->create_tensor(element::f32, Shape{4, 3}); + auto valid_outputs = backend->create_tensor(element::i64, Shape{1}); + + auto backend_boxes = backend->create_tensor(element::f32, boxes_shape); + auto backend_scores = backend->create_tensor(element::f32, scores_shape); + copy_data(backend_boxes, boxes_data); + copy_data(backend_scores, scores_data); + + auto handle = backend->compile(f); + + handle->call({selected_indeces, selected_scores, valid_outputs}, + {backend_boxes, backend_scores}); + + auto selected_indeces_value = read_vector(selected_indeces); + auto selected_scores_value = read_vector(selected_scores); + auto valid_outputs_value = read_vector(valid_outputs); + + std::vector expected_selected_indices = {0, 0, 3, 0, 0, 0, 1, 0, 3, 1, 0, 0}; + std::vector expected_selected_scores = { + 0.0, 0.0, 0.95, 0.0, 0.0, 0.9, 1.0, 0.0, 0.95, 1.0, 0.0, 0.9}; + std::vector expected_valid_outputs = {4}; + + EXPECT_EQ(expected_selected_indices, selected_indeces_value); + EXPECT_EQ(expected_selected_scores, selected_scores_value); + EXPECT_EQ(expected_valid_outputs, valid_outputs_value); +} + +NGRAPH_TEST(${BACKEND_NAME}, nonmaxsuppression_two_classes) +{ + std::vector boxes_data = {0.0, 0.0, 1.0, 1.0, 0.0, 0.1, 1.0, 1.1, + 0.0, -0.1, 1.0, 0.9, 0.0, 10.0, 1.0, 11.0, + 0.0, 10.1, 1.0, 11.1, 0.0, 100.0, 1.0, 101.0}; + + std::vector scores_data = { + 0.9, 0.75, 0.6, 0.95, 0.5, 0.3, 0.9, 0.75, 0.6, 0.95, 0.5, 0.3}; + + const int64_t max_output_boxes_per_class_data = 2; + const float iou_threshold_data = 0.5f; + const float score_threshold_data = 0.0f; + const auto box_encoding = op::v5::NonMaxSuppression::BoxEncodingType::CORNER; + const auto boxes_shape = Shape{1, 6, 4}; + const auto scores_shape = Shape{1, 2, 6}; + + const auto boxes = make_shared(element::f32, boxes_shape); + const auto scores = make_shared(element::f32, scores_shape); + auto max_output_boxes_per_class = + op::Constant::create(element::i64, Shape{}, {max_output_boxes_per_class_data}); + auto iou_threshold = op::Constant::create(element::f32, Shape{}, {iou_threshold_data}); + auto score_threshold = + op::Constant::create(element::f32, Shape{}, {score_threshold_data}); + auto soft_nms_sigma = op::Constant::create(element::f32, Shape{}, {0.0f}); + auto nms = make_shared(boxes, + scores, + max_output_boxes_per_class, + iou_threshold, + score_threshold, + soft_nms_sigma, + box_encoding, + false); + + auto f = make_shared(nms, ParameterVector{boxes, scores}); + + auto backend = runtime::Backend::create("${BACKEND_NAME}"); + + auto selected_indeces = backend->create_tensor(element::i64, Shape{4, 3}); + auto selected_scores = backend->create_tensor(element::f32, Shape{4, 3}); + auto valid_outputs = backend->create_tensor(element::i64, Shape{1}); + + auto backend_boxes = backend->create_tensor(element::f32, boxes_shape); + auto backend_scores = backend->create_tensor(element::f32, scores_shape); + copy_data(backend_boxes, boxes_data); + copy_data(backend_scores, scores_data); + + auto handle = backend->compile(f); + + handle->call({selected_indeces, selected_scores, valid_outputs}, + {backend_boxes, backend_scores}); + + auto selected_indeces_value = read_vector(selected_indeces); + auto selected_scores_value = read_vector(selected_scores); + auto valid_outputs_value = read_vector(valid_outputs); + + std::vector expected_selected_indices = {0, 0, 3, 0, 0, 0, 0, 1, 3, 0, 1, 0}; + std::vector expected_selected_scores = { + 0.0, 0.0, 0.95, 0.0, 0.0, 0.9, 0.0, 1.0, 0.95, 0.0, 1.0, 0.9}; + std::vector expected_valid_outputs = {4}; + + EXPECT_EQ(expected_selected_indices, selected_indeces_value); + EXPECT_EQ(expected_selected_scores, selected_scores_value); + EXPECT_EQ(expected_valid_outputs, valid_outputs_value); +} diff --git a/ngraph/test/runtime/ie/unit_test.manifest b/ngraph/test/runtime/ie/unit_test.manifest index 76e96fc9db5d5b..ea6eb7406c4a5b 100644 --- a/ngraph/test/runtime/ie/unit_test.manifest +++ b/ngraph/test/runtime/ie/unit_test.manifest @@ -1140,6 +1140,26 @@ IE_CPU.onnx_resize11_scales_nearest_asymmetric_floor_dynamic_sizes # Input data precision not supported. Expected float. ctc_greedy_decoder_f16 +# Next nine tests fails in CPU for the following reason. The nGraph function +# for NMS-5 are passed to the method compile() of the backend, but this +# method doesn't apply any nGraph transformations to the passed function, +# and the plugin backend gets CNNNetwork with NMS-5, NMS-5 has dynamic shapes +# for two of three outputs, and results of these two outputs are interpreted +# as scalars. If we apply all needed nGraph transformations to the nGraph +# function with NMS-5 to get the nGraph function with NMSIE3 (internal +# operation, similar with NMS-5, but with all static output shapes), before +# the method compile() call, then tests for INTERPRETER backend for NMS-5 will +# fail, because NMSIE3 has not the reference implementation, but NMS-5 has one. +IE_CPU.nonmaxsuppression_center_point_box_format +IE_CPU.nonmaxsuppression_flipped_coordinates +IE_CPU.nonmaxsuppression_identical_boxes +IE_CPU.nonmaxsuppression_limit_output_size +IE_CPU.nonmaxsuppression_single_box +IE_CPU.nonmaxsuppression_suppress_by_IOU +IE_CPU.nonmaxsuppression_suppress_by_IOU_and_scores +IE_CPU.nonmaxsuppression_two_batches +IE_CPU.nonmaxsuppression_two_classes + #------------------------------------------------------------------------------- # # Inference Engine GPU plugin excludes diff --git a/ngraph/test/runtime/interpreter/int_executable.cpp b/ngraph/test/runtime/interpreter/int_executable.cpp index 2cec9a01f859cb..4efdb29ae5f420 100644 --- a/ngraph/test/runtime/interpreter/int_executable.cpp +++ b/ngraph/test/runtime/interpreter/int_executable.cpp @@ -15,12 +15,15 @@ //***************************************************************************** #include "int_executable.hpp" +#include #include "backend_manager.hpp" #include "ngraph/chrome_trace.hpp" #include "ngraph/except.hpp" #include "ngraph/op/util/op_types.hpp" #include "ngraph/ops.hpp" #include "ngraph/pass/manager.hpp" +#include "ngraph/type/bfloat16.hpp" +#include "ngraph/type/float16.hpp" #include "ngraph/util.hpp" #include "pass/fused_op_decomposition.hpp" #include "pass/liveness.hpp" diff --git a/ngraph/test/runtime/interpreter/int_executable.hpp b/ngraph/test/runtime/interpreter/int_executable.hpp index 0d9e05c6e194a7..0ece39b7909e5e 100644 --- a/ngraph/test/runtime/interpreter/int_executable.hpp +++ b/ngraph/test/runtime/interpreter/int_executable.hpp @@ -69,6 +69,7 @@ #include "ngraph/runtime/reference/max_pool.hpp" #include "ngraph/runtime/reference/min.hpp" #include "ngraph/runtime/reference/negate.hpp" +#include "ngraph/runtime/reference/non_max_suppression.hpp" #include "ngraph/runtime/reference/normalize_l2.hpp" #include "ngraph/runtime/reference/not.hpp" #include "ngraph/runtime/reference/one_hot.hpp" @@ -1303,6 +1304,43 @@ class INTERPRETER_BACKEND_API ngraph::runtime::interpreter::INTExecutable : publ norm->get_eps_mode()); break; } + case OP_TYPEID::NonMaxSuppression_v5: + { + const op::v5::NonMaxSuppression* nms = + static_cast(&node); + + auto info = reference::get_info_for_nms5_evaluation(nms, args); + + std::vector selected_indices(info.out_shape_size); + std::vector selected_scores(info.out_shape_size); + int64_t valid_outputs = 0; + + reference::non_max_suppression(info.boxes_data.data(), + info.boxes_shape, + info.scores_data.data(), + info.scores_shape, + info.max_output_boxes_per_class, + info.iou_threshold, + info.score_threshold, + info.soft_nms_sigma, + selected_indices.data(), + info.out_shape, + selected_scores.data(), + info.out_shape, + &valid_outputs, + info.sort_result_descending); + + auto selected_scores_type = + (args.size() < 4) ? element::f32 : args[3]->get_element_type(); + + reference::nms5_postprocessing(out, + info.output_type, + selected_indices, + selected_scores, + valid_outputs, + selected_scores_type); + break; + } // Fused Ops are not supported in interpreter. They need to be decomposed before execution case OP_TYPEID::DepthToSpace: diff --git a/ngraph/test/runtime/interpreter/opset_int_tbl.hpp b/ngraph/test/runtime/interpreter/opset_int_tbl.hpp index 24f5a5278e5920..8e5ea9f95dc230 100644 --- a/ngraph/test/runtime/interpreter/opset_int_tbl.hpp +++ b/ngraph/test/runtime/interpreter/opset_int_tbl.hpp @@ -65,4 +65,5 @@ NGRAPH_OP(RNNSequence, op::v5) NGRAPH_OP(BatchNormInference, op::v5) NGRAPH_OP(Round, op::v5) NGRAPH_OP(LogSoftmax, op::v5) +NGRAPH_OP(NonMaxSuppression, op::v5) #undef ID_SUFFIX diff --git a/ngraph/test/type_prop/non_max_suppression.cpp b/ngraph/test/type_prop/non_max_suppression.cpp index df8bf1a06d2ac1..ab70f7cb45709b 100644 --- a/ngraph/test/type_prop/non_max_suppression.cpp +++ b/ngraph/test/type_prop/non_max_suppression.cpp @@ -619,43 +619,46 @@ TEST(type_prop, nms_v5_scalar_inputs_check) const auto scores = make_shared(element::f32, Shape{1, 2, 2}); const auto scalar = make_shared(element::f32, Shape{}); - const auto non_scalar = make_shared(element::f32, Shape{1}); + const auto non_0d_or_1d = make_shared(element::f32, Shape{2}); try { - make_shared(boxes, scores, non_scalar, scalar, scalar); + make_shared(boxes, scores, non_0d_or_1d, scalar, scalar); } catch (const NodeValidationFailure& error) { EXPECT_HAS_SUBSTRING(error.what(), - "Expected a scalar for the 'max_output_boxes_per_class' input"); + "Expected 0D or 1D tensor for the 'max_output_boxes_per_class' input"); } try { - make_shared(boxes, scores, scalar, non_scalar, scalar); + make_shared(boxes, scores, scalar, non_0d_or_1d, scalar); } catch (const NodeValidationFailure& error) { - EXPECT_HAS_SUBSTRING(error.what(), "Expected a scalar for the 'iou_threshold' input"); + EXPECT_HAS_SUBSTRING(error.what(), + "Expected 0D or 1D tensor for the 'iou_threshold' input"); } try { - make_shared(boxes, scores, scalar, scalar, non_scalar); + make_shared(boxes, scores, scalar, scalar, non_0d_or_1d); } catch (const NodeValidationFailure& error) { - EXPECT_HAS_SUBSTRING(error.what(), "Expected a scalar for the 'score_threshold' input"); + EXPECT_HAS_SUBSTRING(error.what(), + "Expected 0D or 1D tensor for the 'score_threshold' input"); } try { - make_shared(boxes, scores, scalar, scalar, scalar, non_scalar); + make_shared(boxes, scores, scalar, scalar, scalar, non_0d_or_1d); } catch (const NodeValidationFailure& error) { - EXPECT_HAS_SUBSTRING(error.what(), "Expected a scalar for the 'soft_nms_sigma' input"); + EXPECT_HAS_SUBSTRING(error.what(), + "Expected 0D or 1D tensor for the 'soft_nms_sigma' input"); } } @@ -767,4 +770,4 @@ TEST(type_prop, nms_v5_dynamic_boxes_and_scores) nms->get_output_partial_shape(1).same_scheme(PartialShape{Dimension::dynamic(), 3})); EXPECT_EQ(nms->get_output_shape(2), (Shape{1})); -} \ No newline at end of file +}