Update Proposal Op (#154)

* Proposal reference implementation (openvinotoolkit#3924) * Reference implementation for Proposal, enable CPU SLT * code style fix * add type prop test for invalid anchor count * add unit test * fix shapes in attribute test * temp workaround- disable maring end of boxes list * Disable CPU smoke test- spec misalignment * code style fixes * add some details to the specification * disable myriadx proposal slt * review changes, using usigned int and size_t * improve proposal op shape inference to cover dynamic too, add unit test coverage * remove unused variable in test body * remove batch size in tests where its not used * add post nms topn initialization in tests where it was missing * review comments * style fix * style fix 2 * add tests, remove unused variables, change shape inference checks * style fix * add input tensors type checks and test coverage * align input type in attribute and ngraphreader tests to match specification * fix wrong dimension in error message * proposalv4 ref impl * enable single layer and unit tests for proposalv4 ref impl * align output termination with cpu, enable cpu slt * custom slt compares to detect less-than-predicted number of boxes * custom slt compares to detect less-than-predicted number of boxes * Clarify output termination in spec * review comments * smaller input data for unit tests * add check for batch_dim being static * disable gpu slt for proposal * test data style fix * test data style fix 2 * add type section to specification * shape inference improvement * multiply expected 1st dim in tests by post_nms_topn * add checks and testcases for dynamic ranks * indentation, review comments * reduce code redundancy in ref implementation * remove comment * Fix typo in proposal1 spec * Fix typo in proposal4 spec (cherry picked from commit 210ad81) * Replace nms partial sort to stable sort Co-authored-by: Bartosz Lesniewski <[email protected]>
plaidml · May 25, 2021 · 13b0a23 · 13b0a23
1 parent 2fca665
commit 13b0a23
Show file tree

Hide file tree

Showing 17 changed files with 1,642 additions and 86 deletions.
diff --git a/docs/ops/detection/Proposal_1.md b/docs/ops/detection/Proposal_1.md
@@ -8,7 +8,7 @@
 
 **Detailed description**
 
-*Proposal* has three inputs: a tensor with probabilities whether particular bounding box corresponds to background and foreground, a tensor with bbox_deltas for each of the bounding boxes, a tensor with input image size in the [`image_height`, `image_width`, `scale_height_and_width`] or [`image_height`, `image_width`, `scale_height`, `scale_width`] format. The produced tensor has two dimensions `[batch_size * post_nms_topn, 5]`.
+*Proposal* has three inputs: a tensor with probabilities whether particular bounding box corresponds to background and foreground, a tensor with bbox_deltas for each of the bounding boxes, a tensor with input image size in the [`image_height`, `image_width`, `scale_height_and_width`] or [`image_height`, `image_width`, `scale_height`, `scale_width`] format. The produced tensor has two dimensions `[batch_size * post_nms_topn, 5]`, and for each output box contains batch index and box coordinates.
 *Proposal* layer does the following with the input tensor:
 1.  Generates initial anchor boxes. Left top corner of all boxes is at (0, 0). Width and height of boxes are calculated from *base_size* with *scale* and *ratio* attributes.
 2.  For each point in the first input tensor:
@@ -19,8 +19,9 @@
 5.  Takes top *pre_nms_topn* proposals
 6.  Calculates intersections for boxes and filter out all boxes with \f$intersection/union > nms\_thresh\f$
 7.  Takes top *post_nms_topn* proposals
-8.  Returns top proposals
+8.  Returns top proposals, if there is not enough proposals to fill the whole output tensor, the valid proposals will be terminated with a single -1.
 
+**Attributes**:
 
 * *base_size*
 
@@ -136,15 +137,19 @@
 
 **Inputs**:
 
-*   **1**: 4D input floating point tensor with class prediction scores. Required.
+*   **1**: 4D tensor of type *T* and shape `[batch_size, 2*K, H, W]` with class prediction scores. Required.
 
-*   **2**: 4D input floating point tensor with box bbox_deltas. Required.
+*   **2**: 4D tensor of type *T* and shape `[batch_size, 4*K, H, W]` with deltas for each bounding box. Required.
 
-*   **3**: 1D input floating tensor 3 or 4 elements:  [`image_height`, `image_width`, `scale_height_and_width`] or [`image_height`, `image_width`, `scale_height`, `scale_width`]. Required.
+*   **3**: 1D tensor of type *T* with 3 or 4 elements:  `[image_height, image_width, scale_height_and_width]` or `[image_height, image_width, scale_height, scale_width]`. Required.
 
 **Outputs**:
 
-*   **1**: Floating point tensor of shape `[batch_size * post_nms_topn, 5]`.
+*   **1**: Tensor of type *T* and shape `[batch_size * post_nms_topn, 5]`.
+
+**Types**
+
+* *T*: floating point type.
 
 **Example**
 
@@ -155,4 +160,4 @@
     <input> ... </input>
     <output> ... </output>
 </layer>
-```
+```
diff --git a/docs/ops/detection/Proposal_4.md b/docs/ops/detection/Proposal_4.md
@@ -26,8 +26,11 @@ the second optional tensor of shape `[batch_size * post_nms_topn]` with probabil
 5.  Takes top *pre_nms_topn* proposals
 6.  Calculates intersections for boxes and filter out all boxes with \f$intersection/union > nms\_thresh\f$
 7.  Takes top *post_nms_topn* proposals
-8.  Returns top proposals and optionally their probabilities 
+8.  Returns the results:
+    *   Top proposals, if there is not enough proposals to fill the whole output tensor, the valid proposals will be terminated with a single -1.
+    *   Optionally returns probabilities for each proposal, which are not terminated by any special value.
 
+**Attributes**:
 
 * *base_size*
 

diff --git a/inference-engine/tests/functional/inference_engine/ngraph_reader/proposal_tests.cpp b/inference-engine/tests/functional/inference_engine/ngraph_reader/proposal_tests.cpp
@@ -32,9 +32,9 @@ TEST_F(NGraphReaderTests, ReadProposalNetwork) {
             </output>
         </layer>
         <layer id="2" name="in3" type="Const" version="opset1">
-            <data element_type="i64" offset="0" shape="3" size="24"/>
+            <data element_type="f32" offset="0" shape="3" size="24"/>
             <output>
-                <port id="0" precision="I64">
+                <port id="0" precision="FP32">
                     <dim>3</dim>
                 </port>
             </output>
@@ -85,7 +85,7 @@ TEST_F(NGraphReaderTests, ReadProposalNetwork) {
     std::string model_v6  = R"V0G0N(
 <net name="Network" version="6" batch="1">
     <layers>
-        <layer name="in3" type="Const" precision="I64" id="4">
+        <layer name="in3" type="Const" precision="FP32" id="4">
             <output>
                 <port id="2">
                     <dim>1</dim>
@@ -183,9 +183,9 @@ TEST_F(NGraphReaderTests, ReadProposalNetwork_2) {
             </output>
         </layer>
         <layer id="2" name="in3" type="Const" version="opset1">
-            <data element_type="i64" offset="0" shape="4" size="32"/>
+            <data element_type="f32" offset="0" shape="4" size="32"/>
             <output>
-                <port id="0" precision="I64">
+                <port id="0" precision="FP32">
                     <dim>4</dim>
                 </port>
             </output>
@@ -236,7 +236,7 @@ TEST_F(NGraphReaderTests, ReadProposalNetwork_2) {
     std::string model_v6  = R"V0G0N(
 <net name="Network" version="6" batch="1">
     <layers>
-        <layer name="in3" type="Const" precision="I64" id="4">
+        <layer name="in3" type="Const" precision="FP32" id="4">
             <output>
                 <port id="2">
                     <dim>1</dim>

diff --git a/...engine/tests/functional/plugin/cpu/shared_tests_instances/single_layer_tests/proposal.cpp b/...engine/tests/functional/plugin/cpu/shared_tests_instances/single_layer_tests/proposal.cpp
@@ -45,5 +45,4 @@ INSTANTIATE_TEST_CASE_P(smoke_Proposal_tests, ProposalLayerTest,
                                 ::testing::Values(CommonTestUtils::DEVICE_CPU)),
                         ProposalLayerTest::getTestCaseName
 );
-
 }  // namespace
diff --git a/inference-engine/tests/functional/plugin/gpu/shared_tests_instances/skip_tests_config.cpp b/inference-engine/tests/functional/plugin/gpu/shared_tests_instances/skip_tests_config.cpp
@@ -19,8 +19,11 @@ std::vector<std::string> disabledTestPatterns() {
             R"(.*EltwiseLayerTest.*eltwiseOpType=Pow.*netPRC=I64.*)",
             R"(.*EltwiseLayerTest.*IS=\(.*\..*\..*\..*\..*\).*eltwiseOpType=Pow.*secondaryInputType=CONSTANT.*)",
             // TODO: Issue: 43794
-            R"(.*(PreprocessTest).*(SetScalePreProcess).*)",
-            R"(.*(PreprocessTest).*(ReverseInputChannelsPreProcess).*)",
+            R"(.*(PreprocessTest).*(SetScalePreProcessSetBlob).*)",
+            R"(.*(PreprocessTest).*(SetScalePreProcessGetBlob).*)",
+            R"(.*(PreprocessTest).*(SetMeanValuePreProcessSetBlob).*)",
+            R"(.*(PreprocessTest).*(SetMeanImagePreProcessSetBlob).*)",
+            R"(.*(PreprocessTest).*(ReverseInputChannelsPreProcessGetBlob).*)",
             // TODO: Issue: 41467 -- "unsupported element type f16 op Convert"
             R"(.*(ConvertLayerTest).*targetPRC=FP16.*)",
             // TODO: Issue: 41462
@@ -48,9 +51,13 @@ std::vector<std::string> disabledTestPatterns() {
             R"(.*(LSTMSequence).*mode=CONVERT_TO_TI_RAND_SEQ_LEN.*)",
             R"(.*(smoke_DetectionOutput3In).*)",
             R"(.*(smoke_DetectionOutput5In).*)",
+            // TODO: Issue: 47773
+            R"(.*(ProposalLayerTest).*)",
             R"(.*(ScatterUpdateLayerTest).*)",
 
             // INT8 StridedSlice not supported
             R"(.*(LPT/StridedSliceTransformation).*)",
+            // TODO: Issue: 47219
+            R"(.*DynamicBatchTest.*)",
     };
 }
diff --git a/inference-engine/tests/functional/plugin/myriad/shared_tests_instances/skip_tests_config.cpp b/inference-engine/tests/functional/plugin/myriad/shared_tests_instances/skip_tests_config.cpp
@@ -34,6 +34,8 @@ std::vector<std::string> disabledTestPatterns() {
         R"(.*(DSR_GatherND).*)",
         // TODO: Issue 26090
         ".*DSR_GatherStaticDataDynamicIdx.*f32.*1.3.200.304.*",
+        // TODO: Issue 47315
+        ".*ProposalLayerTest.*",
         // TODO: Issue 46755
         ".*DSR_GatherElements.*"
     };

diff --git a/...ests/functional/shared_test_classes/include/shared_test_classes/single_layer/proposal.hpp b/...ests/functional/shared_test_classes/include/shared_test_classes/single_layer/proposal.hpp
@@ -58,10 +58,44 @@ class ProposalLayerTest
     static std::string getTestCaseName(testing::TestParamInfo<proposalLayerTestParamsSet> obj);
     static std::string SerializeProposalSpecificParams(proposalSpecificParams& params);
     InferenceEngine::Blob::Ptr GenerateInput(const InferenceEngine::InputInfo &info) const override;
+    void Compare(const std::vector<std::vector<std::uint8_t>> &expectedOutputs, const std::vector<InferenceEngine::Blob::Ptr> &actualOutputs) override;
+    template <class T>
+    void Compare(const T *expected, const T *actual, std::size_t size,
+                        T threshold, const std::size_t output_index) {
+        for (std::size_t i = 0; i < size; ++i) {
+            const auto &ref = expected[i];
+            const auto &res = actual[i];
 
+            // verify until first -1 appears in the 1st output.
+            if (output_index == 0 &&
+                CommonTestUtils::ie_abs(ref - static_cast<T>(-1)) <= threshold) {
+                // output0 shape = {x, 5}
+                // output1 shape = {x}
+                // setting the new_size for output1 verification
+                num_selected_boxes = i / 5;
+                return;
+            }
+
+            const auto absoluteDifference = CommonTestUtils::ie_abs(res - ref);
+            if (absoluteDifference <= threshold) {
+                continue;
+            }
+
+            const auto max = std::max(CommonTestUtils::ie_abs(res),
+                                    CommonTestUtils::ie_abs(ref));
+            float diff =
+                static_cast<float>(absoluteDifference) / static_cast<float>(max);
+            ASSERT_TRUE(max != 0 && (diff <= static_cast<float>(threshold)))
+                << "Relative comparison of values expected: " << ref
+                << " and actual: " << res << " at index " << i
+                << " with threshold " << threshold << " failed";
+        }
+    }
 protected:
     void SetUp() override;
-    void Validate() override;
+
+private:
+    size_t num_selected_boxes;
 };
 
 }  // namespace LayerTestsDefinitions
diff --git a/inference-engine/tests/functional/shared_test_classes/src/single_layer/proposal.cpp b/inference-engine/tests/functional/shared_test_classes/src/single_layer/proposal.cpp
@@ -63,9 +63,57 @@ std::string ProposalLayerTest::getTestCaseName(testing::TestParamInfo<proposalLa
     return proposalPramString + result.str();
 }
 
+void ProposalLayerTest::Compare(
+    const std::vector<std::vector<std::uint8_t>> &expectedOutputs,
+    const std::vector<InferenceEngine::Blob::Ptr> &actualOutputs) {
+    num_selected_boxes = 0;
+    for (std::size_t outputIndex = 0; outputIndex < expectedOutputs.size(); ++outputIndex) {
+        const auto &expected = expectedOutputs[outputIndex];
+        const auto &actual = actualOutputs[outputIndex];
+        ASSERT_EQ(expected.size(), actual->byteSize());
+        const auto &expectedBuffer = expected.data();
+
+        auto memory = InferenceEngine::as<InferenceEngine::MemoryBlob>(actual);
+        IE_ASSERT(memory);
+        const auto lockedMemory = memory->rmap();
+        const auto actualBuffer = lockedMemory.as<const std::uint8_t *>();
+
+        const auto &precision = actual->getTensorDesc().getPrecision();
+        auto size = actual->size();
+
+        // verifying the first output if there was less proposals than space
+        // provided,
+        // num_selected_boxes was set, take this into consideration while verifying the 2nd
+        // output
+        if (outputIndex == 1 && num_selected_boxes) {
+            size = num_selected_boxes;
+        }
+
+        switch (precision) {
+        case InferenceEngine::Precision::BF16:
+            Compare(reinterpret_cast<const ngraph::bfloat16 *>(expectedBuffer),
+                    reinterpret_cast<const ngraph::bfloat16 *>(actualBuffer), size,
+                    ngraph::bfloat16(threshold), outputIndex);
+            break;
+        case InferenceEngine::Precision::FP16:
+            Compare(reinterpret_cast<const ngraph::float16 *>(expectedBuffer),
+                    reinterpret_cast<const ngraph::float16 *>(actualBuffer), size,
+                    ngraph::float16(threshold), outputIndex);
+            break;
+        case InferenceEngine::Precision::FP32:
+            Compare<float>(reinterpret_cast<const float *>(expectedBuffer),
+                            reinterpret_cast<const float *>(actualBuffer), size,
+                            threshold, outputIndex);
+            break;
+        default:
+        FAIL() << "Comparator for " << precision << " precision isn't supported";
+    }
+  }
+}
+
 void ProposalLayerTest::SetUp() {
     proposalSpecificParams proposalParams;
-    std::vector<float> img_info = {225.0f, 225.0f, 1.0f};
+    std::vector<float> img_info = {3.0f, 3.0f, 1.0f};
 
     std::tie(proposalParams, targetDevice) = this->GetParam();
     base_size_type base_size;
@@ -98,10 +146,11 @@ void ProposalLayerTest::SetUp() {
     std::vector<size_t> imageInfoShape = {3};
 
     auto ngPrc = FuncTestUtils::PrecisionUtils::convertIE2nGraphPrc(InferenceEngine::Precision::FP16);
-    auto params = ngraph::builder::makeParams(ngPrc, {{"scores", scoresShape}, {"boxes", boxesShape}});
+        // a_ and b_ are a workaround to solve alphabetic param sorting that destroys ordering
+    auto params = ngraph::builder::makeParams(ngPrc, {{"a_scores", scoresShape}, {"b_boxes", boxesShape}});
     auto paramOuts = ngraph::helpers::convert2OutputVector(ngraph::helpers::castOps2Nodes<ngraph::op::Parameter>(params));
 
-    auto proposal = std::dynamic_pointer_cast<ngraph::opset1::Proposal>(
+    auto proposal = std::dynamic_pointer_cast<ngraph::opset4::Proposal>(
              ngraph::builder::makeProposal(paramOuts[0], paramOuts[1], img_info, ngPrc,
                                            base_size,
                                            pre_nms_topn,
@@ -118,23 +167,22 @@ void ProposalLayerTest::SetUp() {
                                            box_coordinate_scale,
                                            framework));
 
-    ngraph::ResultVector results{std::make_shared<ngraph::opset1::Result>(proposal)};
+    ngraph::ResultVector results{
+        std::make_shared<ngraph::opset1::Result>(proposal->output(0)),
+        std::make_shared<ngraph::opset1::Result>(proposal->output(1))};
     function = std::make_shared<ngraph::Function>(results, params, "proposal");
 }
 
 InferenceEngine::Blob::Ptr ProposalLayerTest::GenerateInput(const InferenceEngine::InputInfo &info) const {
     InferenceEngine::Blob::Ptr blobPtr;
 
     const std::string name = info.name();
-    if (name == "scores") {
+    if (name == "a_scores") {
         blobPtr = FuncTestUtils::createAndFillBlobFloat(info.getTensorDesc(), 1, 0, 1000, 8234231);
-    } else if (name == "boxes") {
+    } else if (name == "b_boxes") {
         blobPtr = FuncTestUtils::createAndFillBlobFloatNormalDistribution(info.getTensorDesc(), 0.0f, 0.2f, 7235346);
     }
 
     return blobPtr;
 }
-
-// TODO: for validation, reference version is required (#28373)
-void ProposalLayerTest::Validate() {}
 }  // namespace LayerTestsDefinitions
diff --git a/inference-engine/tests/ngraph_helpers/ngraph_functions/src/proposal.cpp b/inference-engine/tests/ngraph_helpers/ngraph_functions/src/proposal.cpp
@@ -44,10 +44,11 @@ std::shared_ptr<Node> makeProposal(const ngraph::Output<Node> &class_probs,
     attrs.box_size_scale = box_size_scale;
     attrs.box_coordinate_scale = box_coordinate_scale;
     attrs.framework = framework;
+    attrs.infer_probs = true;
 
     auto image_shape = makeConstant(ngraph::element::Type_t::f32, {3}, image_info);
 
-    return std::make_shared<opset1::Proposal>(class_probs, class_logits, image_shape, attrs);
+    return std::make_shared<opset4::Proposal>(class_probs, class_logits, image_shape, attrs);
 }
 
 }  // namespace builder