Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for custom ONNX operator PriorBoxClustered #4202

Merged
merged 16 commits into from
Feb 15, 2021
Merged

Add support for custom ONNX operator PriorBoxClustered #4202

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
f9db1e6
Add support for custom ONNX operator PriorBoxClustered
Feb 5, 2021
b3f48e1
Fix codestyle
Feb 5, 2021
ff9263f
Add unittest for constant input
Feb 8, 2021
d53d732
Remove commented code
Feb 8, 2021
c96c73b
Add priorboxclustered to legacy_ops_to_fixup
Feb 8, 2021
f917ca1
Disable failing tests on cpu
Feb 8, 2021
282cb6e
Fix implementation of priorboxclustered op
Feb 9, 2021
5bb3909
Update unittests
Feb 9, 2021
fe6ef05
Merge branch 'master' of https://github.com/openvinotoolkit/openvino …
Feb 9, 2021
7a6757d
Move custom onnx op priorboxclustered to priorbox context
Feb 9, 2021
a446adf
Merge branch 'master' of https://github.com/openvinotoolkit/openvino …
Feb 9, 2021
b98b8b5
Move priorboxClustered tests to org_openvino specfic test file
Feb 9, 2021
79ce775
Remove redundant using clause
Feb 9, 2021
de0ec1a
Address requested changes
Feb 10, 2021
b0c8246
Fix one of test models
Feb 10, 2021
6d53796
Merge branch 'master' of https://github.com/openvinotoolkit/openvino …
Feb 11, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ngraph/frontend/onnx_import/src/core/transform.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ namespace ngraph
"GroupNorm",
"Normalize",
"PriorBox",
"PriorBoxClustered",
"Swish"};

/// \brief Add support for models with custom operators mistakenly registered in
Expand Down
29 changes: 29 additions & 0 deletions ngraph/frontend/onnx_import/src/op/org.openvinotoolkit/prior_box.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,35 @@ namespace ngraph
axes)};
}

OutputVector prior_box_clustered(const Node& node)
{
auto inputs = node.get_ng_inputs();
NGRAPH_CHECK(inputs.size() == 2, "Invalid number of inputs");

auto output_shape = std::make_shared<default_opset::ShapeOf>(inputs[0]);
auto image_shape = std::make_shared<default_opset::ShapeOf>(inputs[1]);
auto output_shape_slice = detail::make_slice(output_shape, 2, 4);
auto image_shape_slice = detail::make_slice(image_shape, 2, 4);
bsledz marked this conversation as resolved.
Show resolved Hide resolved

ngraph::op::PriorBoxClusteredAttrs attrs{};
attrs.widths = node.get_attribute_value<std::vector<float>>("width", {1.0});
attrs.heights = node.get_attribute_value<std::vector<float>>("height", {1.0});
bsledz marked this conversation as resolved.
Show resolved Hide resolved
attrs.clip = static_cast<bool>(node.get_attribute_value<int64_t>("clip", 0));
bsledz marked this conversation as resolved.
Show resolved Hide resolved
attrs.variances =
node.get_attribute_value<std::vector<float>>("variance", {0.1f});
attrs.step_heights = node.get_attribute_value<float>("step_h", 0.0f);
attrs.step_widths = node.get_attribute_value<float>("step_w", 0.0f);
attrs.offset = node.get_attribute_value<float>("offset", 0.0f);

auto axes = default_opset::Constant::create(
element::i64, Shape{1}, std::vector<int64_t>{0});

return {std::make_shared<default_opset::Unsqueeze>(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this Unsqueeze needed here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added this Unsqueeze to match pattern from: ConvertPriorBoxClusteredToLegacy transformation.
I'm not sure what is the purpose of the Unsqueeze there, @itikhono, could you please help us?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The semantics of the custom PriorBox (Clustered) operator in ONNX and the specified version of the op in OV are different.
We use this sub-graph to convert ONNX PriorBox to the spec version (Unsqueeze is required)
image
But our plugins support only ONNX version, so we try to detect exactly this pattern and convert it from our specified version to supported by plugins using ConvertPriorBoxClusteredToLegacy.
Note: This code should be the same as PriorboxMutation transformation in MO. Please double-check it.

std::make_shared<default_opset::PriorBoxClustered>(
output_shape_slice, image_shape_slice, attrs),
axes)};
}

} // namespace set_1

} // namespace op
Expand Down
2 changes: 2 additions & 0 deletions ngraph/frontend/onnx_import/src/op/org.openvinotoolkit/prior_box.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ namespace ngraph
{
OutputVector prior_box(const Node& node);

OutputVector prior_box_clustered(const Node& node);

} // namespace set_1

} // namespace op
Expand Down
2 changes: 2 additions & 0 deletions ngraph/frontend/onnx_import/src/ops_bridge.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -493,6 +493,8 @@ namespace ngraph
REGISTER_OPERATOR_WITH_DOMAIN(OPENVINO_ONNX_DOMAIN, "GroupNorm", 1, group_norm);
REGISTER_OPERATOR_WITH_DOMAIN(OPENVINO_ONNX_DOMAIN, "Normalize", 1, normalize);
REGISTER_OPERATOR_WITH_DOMAIN(OPENVINO_ONNX_DOMAIN, "PriorBox", 1, prior_box);
REGISTER_OPERATOR_WITH_DOMAIN(
OPENVINO_ONNX_DOMAIN, "PriorBoxClustered", 1, prior_box_clustered);
REGISTER_OPERATOR_WITH_DOMAIN(OPENVINO_ONNX_DOMAIN, "Swish", 1, swish);
}

Expand Down
122 changes: 122 additions & 0 deletions ngraph/test/models/onnx/priorbox_clustered.prototxt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
ir_version: 3
producer_name: "nGraph ONNX Importer"
graph {
node {
domain: "org.openvinotoolkit"
input: "A"
input: "B"
output: "out"
op_type: "PriorBoxClustered"
attribute {
name: "width"
floats: 0.1
floats: 0.1
floats: 0.2
floats: 0.2
type: FLOATS
}
attribute {
name: "height"
floats: 0.1
floats: 0.1
floats: 0.2
floats: 0.2
type: FLOATS
}
attribute {
name: "step_w"
f: 64
type: FLOAT
}
attribute {
name: "clip"
i: 1
type: INT
}
attribute {
name: "step_h"
f: 64
type: FLOAT
}
attribute {
name: "offset"
f: 0.5
type: FLOAT
}
attribute {
name: "variance"
floats: 0.1
floats: 0.1
floats: 0.2
floats: 0.2
type: FLOATS
}
}
name: "compute_graph"
input {
name: "A"
type {
tensor_type {
elem_type: 1
shape {
dim {
dim_value: 1
}
dim {
dim_value: 1
}
dim {
dim_value: 1
}
dim {
dim_value: 1
}
}
}
}
}
input {
name: "B"
type {
tensor_type {
elem_type: 1
shape {
dim {
dim_value: 1
}
dim {
dim_value: 1
}
dim {
dim_value: 1
}
dim {
dim_value: 1
}
}
}
}
}
output {
name: "out"
type {
tensor_type {
elem_type: 1
shape {
dim {
dim_value: 1
}
dim {
dim_value: 2
}
dim {
dim_value: 4
}
}
}
}
}
}
opset_import {
version: 1
}
86 changes: 86 additions & 0 deletions ngraph/test/models/onnx/priorbox_clustered_most_attrs_default.prototxt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
ir_version: 3
producer_name: "nGraph ONNX Importer"
graph {
node {
domain: "org.openvinotoolkit"
input: "A"
input: "B"
output: "out"
op_type: "PriorBoxClustered"
attribute {
name: "variance"
floats: 0.1
floats: 0.1
floats: 0.2
floats: 0.2
type: FLOATS
}
}
name: "compute_graph"
input {
name: "A"
type {
tensor_type {
elem_type: 1
shape {
dim {
dim_value: 1
}
dim {
dim_value: 1
}
dim {
dim_value: 2
}
dim {
dim_value: 1
}
}
}
}
}
input {
name: "B"
type {
tensor_type {
elem_type: 1
shape {
dim {
dim_value: 1
}
dim {
dim_value: 1
}
dim {
dim_value: 3
}
dim {
dim_value: 3
}
}
}
}
}
output {
name: "out"
type {
tensor_type {
elem_type: 1
shape {
dim {
dim_value: 1
}
dim {
dim_value: 2
}
dim {
dim_value: 8
}
}
}
}
}
}
opset_import {
version: 1
}
52 changes: 51 additions & 1 deletion ngraph/test/onnx/onnx_import_org_openvino.in.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,56 @@ NGRAPH_TEST(${BACKEND_NAME}, onnx_prior_box)
test_case.run();
}

NGRAPH_TEST(${BACKEND_NAME}, onnx_priorbox_clustered)
{
auto function = onnx_import::import_onnx_model(
file_util::path_join(SERIALIZED_ZOO, "onnx/priorbox_clustered.prototxt"));

auto test_case = test::TestCase<TestEngine, test::TestCaseType::DYNAMIC>(function);
std::vector<float> A{15.0};
std::vector<float> B{10.0};
std::vector<float> output = {
1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0,
0.1, 0.1, 0.2, 0.2, 0.1, 0.1, 0.2, 0.2, 0.1, 0.1, 0.2, 0.2, 0.1, 0.1, 0.2, 0.2,
};
test_case.add_input<float>(A);
test_case.add_input<float>(B);
test_case.add_expected_output<float>(Shape{1, 2, 16}, output);
test_case.run();
}

NGRAPH_TEST(${BACKEND_NAME}, onnx_priorbox_clustered_most_attrs_default)
{
auto function = onnx_import::import_onnx_model(file_util::path_join(
SERIALIZED_ZOO, "onnx/priorbox_clustered_most_attrs_default.prototxt"));

auto test_case = test::TestCase<TestEngine, test::TestCaseType::DYNAMIC>(function);
std::vector<float> A(1 * 1 * 2 * 1);
std::iota(std::begin(A), std::end(A), 0.0f);
std::vector<float> B(1 * 1 * 3 * 3);
std::iota(std::begin(B), std::end(B), 0.0f);
std::vector<float> output = {-0.1666666716337203979,
-0.1666666716337203979,
0.1666666716337203979,
0.1666666716337203979,
-0.1666666716337203979,
0.3333333432674407959,
0.1666666716337203979,
0.6666666865348815918,
0.1,
0.1,
0.2,
0.2,
0.1,
0.1,
0.2,
0.2};
test_case.add_input<float>(A);
test_case.add_input<float>(B);
test_case.add_expected_output<float>(Shape{1, 2, 8}, output);
test_case.run();
}

NGRAPH_TEST(${BACKEND_NAME}, onnx_detection_output)
{
const auto function = onnx_import::import_onnx_model(
Expand Down Expand Up @@ -490,4 +540,4 @@ NGRAPH_TEST(${BACKEND_NAME}, onnx_model_experimental_detectron_topk_rios)

test_case.add_expected_output<float>(Shape{1, 4}, {1, 1, 3, 4});
test_case.run();
}
}