Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Object Detection] Gluoncv SSD support on CPU #2353

Merged
merged 43 commits into from
Mar 11, 2019
Merged
Changes from 1 commit
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
1da5012
Refactor NMS
Jul 29, 2018
1eb27a8
Avoid using function call for hybrid frontend
Sep 4, 2018
1e2cdb5
Modify nms
Nov 28, 2018
d892867
Add box_nms
Dec 11, 2018
c19526b
Add test for get_valid_counts
Dec 13, 2018
9e0eee7
Add missing operators
Dec 16, 2018
037ca23
Add id_index to box_nms op
Dec 16, 2018
8f9e9e2
Add l2_normalize to from_mxnet
Dec 16, 2018
5c5e6f7
Modify SSD tutorial
Dec 19, 2018
3054af5
Fix tutorial
Dec 19, 2018
ef21b02
Relay support
Jan 8, 2019
31f0927
Add more ops to from_mxnet
Jan 8, 2019
fb43612
Support multibox op with hybrid script
Jan 11, 2019
717e61c
Fix slice_axis relay register issue
Jan 12, 2019
b9681ee
Add get_valid_counts relay test
Jan 13, 2019
e277a55
Fix multibox_transform_loc
Jan 14, 2019
ef00b7f
Fix relay from_mxnet
Jan 14, 2019
fa89a2a
Fix l2_normalize
Jan 14, 2019
925c140
Fix lint
Jan 15, 2019
6819dc3
Add cuda schedule for get_valid_counts
Jan 15, 2019
8eaff5c
Fix tutorial
Jan 15, 2019
7423762
Fix ctx_list
Jan 18, 2019
b3c8a7c
Add install gluoncv
Jan 18, 2019
986c4f7
Disable box_nms in frontend test
Jan 28, 2019
e7df94c
Fix test get_valid_counts numpy result
Jan 28, 2019
45b6aac
Rename ssd tutorial
Jan 31, 2019
26ece34
Fix rebase
Jan 31, 2019
2f8aef8
Refactor nms
Feb 19, 2019
d30be86
Rollback nnvm
Feb 19, 2019
6b1fd7a
Refactor relay nms
Feb 20, 2019
42571cf
Add max_output_size arg
Feb 22, 2019
f8fecec
Make return_indices optional
Feb 24, 2019
11c8bba
Minor fix
Feb 24, 2019
908eedb
Resolve conflict
Feb 24, 2019
9743f15
Rename topk to top_k
Feb 25, 2019
4bd6fec
Fix example code
Feb 27, 2019
d1e95f9
Fix lint
Mar 3, 2019
292130d
Minor fix
Mar 3, 2019
fb94ffe
Remove contrib_slice_axis
Mar 4, 2019
5c0cee9
Resolve conflict
Mar 4, 2019
28d479a
Address minor comments
Mar 5, 2019
c2e02e4
Move tutorial
Mar 7, 2019
d20024c
Resolve conflict
Mar 8, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Relay support
Wang authored and kevinthesun committed Mar 11, 2019
commit ef21b022fb4fef9c58ddb18e4e6ad9cb8c74b774
15 changes: 15 additions & 0 deletions include/tvm/relay/attrs/transform.h
Original file line number Diff line number Diff line change
@@ -171,6 +171,21 @@ struct StridedSliceAttrs : public tvm::AttrsNode<StridedSliceAttrs> {
}
};

struct SliceAxisAttrs : public tvm::AttrsNode<SliceAxisAttrs> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we already have a slice op. Let us just use that one to avoid duplication:

see https://docs.tvm.ai/langref/relay_op.html#tvm.relay.strided_slice

Copy link
Contributor Author

@kevinthesun kevinthesun Feb 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem of directly using strided_slice to generate slice_axis in the frontend converter is we need to know the input shape to generate arguments for strided_slice. This is the major task in compute of slice_axis. After this is done, we just call strided_slice.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we only need to know the input dimension instead of the shape?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, input dimension.

int axis;
int begin;
int end;

TVM_DECLARE_ATTRS(SliceAxisAttrs, "relay.attrs.SliceAxisAttrs") {
TVM_ATTR_FIELD(axis)
.describe("Axis along which to be sliced.");
TVM_ATTR_FIELD(begin)
.describe("Index for begin of slice");
TVM_ATTR_FIELD(end).set_default(0)
.describe("Index for end of the slice");
}
};

struct SliceLikeAttrs : public tvm::AttrsNode<SliceLikeAttrs> {
Array<Integer> axes;

32 changes: 24 additions & 8 deletions include/tvm/relay/attrs/vision.h
Original file line number Diff line number Diff line change
@@ -58,19 +58,35 @@ struct MultiBoxTransformLocAttrs
}
};

/*! \brief Attributes used in non_maximum_suppression operators */
/*! \brief Attributes used in get_valid_counts operator */
struct GetValidCountsAttrs : public tvm::AttrsNode<GetValidCountsAttrs>{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: public tvm::AttrsNode {

space

double score_threshold;

TVM_DECLARE_ATTRS(GetValidCountsAttrs, "relay.attrs.GetValidCountsAttrs") {
TVM_ATTR_FIELD(score_threshold).set_default(0.0)
.describe("Lower limit of score for valid bounding boxes.");
}
};

/*! \brief Attributes used in non_maximum_suppression operator */
struct NMSAttrs : public tvm::AttrsNode<NMSAttrs>{
double overlap_threshold;
double iou_threshold;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let us deliberate on the choice of the parameter name. In particular why the name

  • iou_threshold, id_index(while it refers to axis?), do_rearrange

Copy link
Contributor Author

@kevinthesun kevinthesun Feb 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

iou_threshold is the threshold of intersection over union(IoU). This is different from mxnet API, but I think it might be easier to understand in object detection context? @zhreshold

You are right. id_index in fact means axis. I think we can change it to id_axis.

do_rearrange is whether to move all invalid boxed to the bottom. Maybe we can change it to invalid_to_bottom to be more meaningful?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not have a strong opinion on the naming choices, but perhaps it would be helpful to bring several people who are involved. Mainly because API relates to backward compatibility and we want to be careful. How about a quick RFC?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bool force_suppress;
int topk;
int id_index;
bool do_rearrange;

TVM_DECLARE_ATTRS(NMSAttrs, "relay.attrs.NMSAttrs") {
TVM_ATTR_FIELD(overlap_threshold).set_default(0.5)
.describe("Non-maximum suppression threshold.");
TVM_ATTR_FIELD(force_suppress).set_default(false)
.describe("Suppress all detections regardless of class_id.");
TVM_ATTR_FIELD(topk).set_default(-1)
.describe("Keep maximum top k detections before nms, -1 for no limit.");
TVM_ATTR_FIELD(iou_threshold).set_default(0.5)
.describe("Non-maximum suppression threshold.");
TVM_ATTR_FIELD(force_suppress).set_default(false)
.describe("Suppress all detections regardless of class_id.");
TVM_ATTR_FIELD(topk).set_default(-1)
.describe("Keep maximum top k detections before nms, -1 for no limit.");
TVM_ATTR_FIELD(id_index).set_default(0)
.describe("Axis index of id.");
TVM_ATTR_FIELD(do_rearrange).set_default(false)
.describe("Whether to move all valid bounding boxes to the top.");
}
};

18 changes: 9 additions & 9 deletions nnvm/include/nnvm/top/nn.h
Original file line number Diff line number Diff line change
@@ -402,9 +402,9 @@ struct LayoutTransformParam : public dmlc::Parameter<LayoutTransformParam> {

DMLC_DECLARE_PARAMETER(LayoutTransformParam) {
DMLC_DECLARE_FIELD(src_layout).set_default("__undef__")
.describe("Dimension ordering of data");
.describe("Dimension ordering of data");
DMLC_DECLARE_FIELD(dst_layout).set_default("__undef__")
.describe("Dimension ordering of data.");
.describe("Dimension ordering of data.");
}
};

@@ -419,13 +419,13 @@ struct MultiBoxPriorParam : public dmlc::Parameter<MultiBoxPriorParam> {
DMLC_DECLARE_FIELD(sizes).set_default(Tuple<float>({1.0}))
.describe("List of sizes of generated MultiBoxPriores.");
DMLC_DECLARE_FIELD(ratios).set_default(Tuple<float>({1.0}))
.describe("List of aspect ratios of generated MultiBoxPriores.");
.describe("List of aspect ratios of generated MultiBoxPriores.");
DMLC_DECLARE_FIELD(steps).set_default(Tuple<float>({-1.0, -1.0}))
.describe("Priorbox step across y and x, -1 for auto calculation.");
.describe("Priorbox step across y and x, -1 for auto calculation.");
DMLC_DECLARE_FIELD(offsets).set_default(Tuple<float>({0.5, 0.5}))
.describe("Priorbox center offsets, y and x respectively.");
.describe("Priorbox center offsets, y and x respectively.");
DMLC_DECLARE_FIELD(clip).set_default(false)
.describe("Whether to clip out-of-boundary boxes.");
.describe("Whether to clip out-of-boundary boxes.");
}
};

@@ -461,11 +461,11 @@ struct NMSParam : public dmlc::Parameter<NMSParam> {
DMLC_DECLARE_FIELD(iou_threshold).set_default(0.5)
.describe("Non-maximum suppression threshold.");
DMLC_DECLARE_FIELD(force_suppress).set_default(false)
.describe("Suppress all detections regardless of class_id.");
.describe("Suppress all detections regardless of class_id.");
DMLC_DECLARE_FIELD(topk).set_default(-1)
.describe("Keep maximum top k detections before nms, -1 for no limit.");
DMLC_DECLARE_FIELD(id_index).set_default(0)
.describe("Keep maximum top k detections before nms, -1 for no limit.");
DMLC_DECLARE_FIELD(id_index).set_default(0)
.describe("Axis index for id.");
DMLC_DECLARE_FIELD(do_rearrange).set_default(false)
.describe("Whether to move all valid bounding boxes to the top.");
}
1 change: 0 additions & 1 deletion nnvm/src/top/vision/nms.cc
Original file line number Diff line number Diff line change
@@ -11,7 +11,6 @@
#include <nnvm/op_attr_types.h>
#include <nnvm/compiler/op_attr_types.h>
#include "../op_common.h"
#include "../elemwise_op_common.h"

namespace nnvm {
namespace top {
10 changes: 1 addition & 9 deletions nnvm/tests/python/compiler/test_top_level4.py
Original file line number Diff line number Diff line change
@@ -700,16 +700,8 @@ def test_slice_like():
def verify_slice_axis(dshape, axis, begin, end):
data = sym.Variable("data")
net = sym.slice_axis(data, axis=axis, begin=begin, end=end)
if axis < 0:
axis += len(dshape)
if begin < 0:
begin += dshape[axis]
if end <= 0:
end += dshape[axis]
np_data = np.random.uniform(size=dshape)
slc = [slice(None)] * len(dshape)
slc[axis] = slice(begin, end)
np_out = np_data[slc]
np_out = topi.testing.slice_axis_python(np_data, axis, begin, end)

dtype = "float32"
for target, ctx in ctx_list():
13 changes: 12 additions & 1 deletion nnvm/tests/python/frontend/mxnet/test_forward.py
Original file line number Diff line number Diff line change
@@ -227,6 +227,7 @@ def test_forward_slice():
mx_sym = mx.sym.slice(data, begin=(-1, 1), end=(-3, 4), step=(-1, 2))
verify_mxnet_frontend_impl(mx_sym, (3, 4), (2, 2))

<<<<<<< HEAD
def test_forward_maximum():
a = mx.sym.var('a')
b = mx.sym.var('b')
@@ -289,6 +290,15 @@ def test_forward_minimum():
tvm_out = m.get_output(0, tvm.nd.empty(out_shape, dtype)).asnumpy()
tvm.testing.assert_allclose(mx_out, tvm_out, rtol=1e-5, atol=1e-5)

def test_forward_slice_axis():
data = mx.sym.var('data')
mx_sym = mx.sym.slice_axis(data, axis=1, begin=-5)
verify_mxnet_frontend_impl(mx_sym, (1, 10, 6), (1, 5, 6))

def test_forward_l2_normalize():
data = mx.sym.var('data')
mx_sym = mx.sym.L2Normalization(data, mode="channel")
verify_mxnet_frontend_impl(mx_sym, (2, 3, 4, 5), (2, 3, 4, 5))

if __name__ == '__main__':
test_forward_mlp()
@@ -315,4 +325,5 @@ def test_forward_minimum():
test_forward_slice()
test_forward_maximum()
test_forward_minimum()

test_forward_slice_axis()
test_forward_l2_normalize()
43 changes: 43 additions & 0 deletions python/tvm/relay/frontend/mxnet.py
Original file line number Diff line number Diff line change
@@ -380,6 +380,47 @@ def _mx_proposal(inputs, attrs):
return _op.vision.proposal(inputs[0], inputs[1], inputs[2], **new_attrs)


def _mx_box_nms(inputs, attrs):
force_suppress = attrs.get_bool("force_suppress", False)
overlap_thresh = attrs.get_float('overlap_thresh', 0.5)
topk = attrs.get_int('topk', -1)
valid_thresh = attrs.get_float('valid_thresh', 0)
coord_start = attrs.get_int('coord_start', 2)
score_index = attrs.get_int('score_index', 1)
id_index = attrs.get_int('id_index', -1)
in_format = attrs.get_str('in_format', 'corner')
out_format = attrs.get_str('out_format', 'corner')
if coord_start != 2:
raise RuntimeError('coord_start %s is not supported.' % coord_start)
if score_index != 1:
raise RuntimeError('score_index %s is not supported.' % score_index)
if id_index != -1 and int(id_index) != 0:
raise RuntimeError('id_index %s is not supported.' % id_index)
if in_format != 'corner':
raise RuntimeError('in_format %s is not supported.' % in_format)
if out_format != 'corner':
raise RuntimeError('out_format %s is not supported.' % out_format)

valid_counts, inter_out = \
_op.vision.get_valid_counts(inputs[0], score_threshold=valid_thresh)
nms_out = _op.vision.nms(inter_out, valid_counts,
iou_threshold=overlap_thresh,
force_suppress=force_suppress,
topk=topk, id_index=id_index,
do_rearrange=True)
return nms_out


def _mx_l2_normalize(inputs, attrs):
new_attrs = {}
mode = attrs.get_str('mode', 'instance')
if mode != 'channel':
raise RuntimeError('mode %s is not supported.' % mode)
new_attrs['eps'] = attrs.get_float('eps', 1e-10)
new_attrs['axis'] = 1
return _op.nn.l2_normalize(inputs[0], **new_attrs)


# Note: due to attribute conversion constraint
# ops in the identity set must be attribute free
_identity_list = [
@@ -481,7 +522,9 @@ def _mx_proposal(inputs, attrs):
"slice" : _mx_slice,
"slice_like" : _mx_slice_like,
"slice_axis" : _mx_slice_axis,
"L2Normalization" : _mx_l2_normalize,∂
"SliceChannel" : _mx_split,
"slice_axis" : _mx_slice_axis,
"split" : _mx_split,
"expand_dims" : _mx_expand_dims,
"Concat" : _mx_concat,
1 change: 1 addition & 0 deletions python/tvm/relay/op/_transform.py
Original file line number Diff line number Diff line change
@@ -21,6 +21,7 @@
_reg.register_schedule("arange", schedule_injective)
_reg.register_schedule("cast", schedule_injective)
_reg.register_schedule("strided_slice", schedule_injective)
_reg.register_schedule("slice_axis", schedule_injective)
_reg.register_schedule("slice_like", schedule_injective)
_reg.register_schedule("split", schedule_injective)
_reg.register_schedule("take", schedule_injective)
28 changes: 27 additions & 1 deletion python/tvm/relay/op/transform.py
Original file line number Diff line number Diff line change
@@ -456,7 +456,7 @@ def strided_slice(data, begin, end, strides=None):
The indices to begin with in the slicing.

end: list of int
Indicies indicating end of the slice.
Indices indicating end of the slice.

strides: list of int, optional
Specifies the stride values, it can be negative in that case,
@@ -471,6 +471,32 @@ def strided_slice(data, begin, end, strides=None):
return _make.strided_slice(data, list(begin), list(end), list(strides))


def slice_axis(data, axis, begin, end=None):
"""Slice input array along specific axis.

Parameters
----------
data : relay.Expr
The source array to be sliced.

axis : int
Axis to be sliced.

begin: int
The index to begin with in the slicing.

end: int, optional
The index indicating end of the slice.

Returns
-------
ret : relay.Expr
The computed result.
"""
end = end or 0
return _make.slice_axis(data, axis, begin, end)


def slice_like(data, shape_like, axes=None):
"""Slice the first input with respect to the second input.

2 changes: 1 addition & 1 deletion python/tvm/relay/op/vision/__init__.py
Original file line number Diff line number Diff line change
@@ -6,6 +6,6 @@
from .nms import *
from .rcnn import *
from .yolo import *
from . import _multibox
from . import _rcnn
from . import _yolo
from . import _vision
Original file line number Diff line number Diff line change
@@ -54,6 +54,23 @@ def compute_multibox_transform_loc(attrs, inputs, _, target):
reg.register_pattern("vision.multibox_detection", OpPattern.OPAQUE)


# Get counts of valid boxes
@reg.register_schedule("vision.get_valid_counts")
def schedule_get_valid_counts(_, outs, target):
"""Schedule definition of get_valid_counts"""
with target:
return topi.generic.schedule_nms(outs)


@reg.register_compute("vision.get_valid_counts")
def compute_get_valid_counts(attrs, inputs, _, target):
"""Compute definition of get_valid_counts"""
score_threshold = get_const_float(attrs.score_threshold)
return topi.vision.get_valid_counts(inputs[0], score_threshold)

reg.register_pattern("vision.get_valid_counts", OpPattern.OPAQUE)


# non-maximum suppression
@reg.register_schedule("vision.nms")
def schedule_nms(_, outs, target):
@@ -65,12 +82,14 @@ def schedule_nms(_, outs, target):
@reg.register_compute("vision.nms")
def compute_nms(attrs, inputs, _, target):
"""Compute definition of nms"""
overlap_threshold = get_const_float(attrs.overlap_threshold)
iou_threshold = get_const_float(attrs.iou_threshold)
force_suppress = bool(get_const_int(attrs.force_suppress))
topk = get_const_int(attrs.topk)
id_index = get_const_int(attrs.id_index)
do_rearrange = bool(get_const_int(attrs.do_rearrange))
return [
topi.vision.nms(inputs[0], inputs[1], overlap_threshold,
force_suppress, topk)
topi.vision.nms(inputs[0], inputs[1], iou_threshold,
force_suppress, topk, id_index, do_rearrange)
]


41 changes: 37 additions & 4 deletions python/tvm/relay/op/vision/nms.py
Original file line number Diff line number Diff line change
@@ -2,11 +2,37 @@
from __future__ import absolute_import as _abs
from . import _make

def get_valid_counts(data,
score_threshold):
"""Get valid count of bounding boxes given a score threshold.
Also moves valid boxes to the top of input data.

Parameters
----------
data : relay.Expr
Input data. 3-D tensor with shape [batch_size, num_anchors, 6].

score_threshold : optional, float
Lower limit of score for valid bounding boxes.

Returns
-------
out_tensor : relay.Expr
Rearranged data tensor.

valid_count : relay.Expr
1-D tensor for valid number of boxes.
"""
return _make.get_valid_counts(data, score_threshold)


def nms(data,
valid_count,
overlap_threshold=0.5,
iou_threshold=0.5,
force_suppress=False,
topk=-1):
topk=-1,
id_index=0,
do_rearrange=False):
"""Non-maximum suppression operator for object detection.

Parameters
@@ -19,7 +45,7 @@ def nms(data,
valid_count : relay.Expr
1-D tensor for valid number of boxes.

overlap_threshold : float, optional
iou_threshold : float, optional
Non-maximum suppression threshold.

force_suppress : bool, optional
@@ -28,9 +54,16 @@ def nms(data,
topk : int, optional
Keep maximum top k detections before nms, -1 for no limit.

id_index : optional, int
index of the class categories, -1 to disable.

do_rearrange : optional, boolean
Whether to move all valid bounding boxes to the top.

Returns
-------
out : relay.Expr
3-D tensor with shape [batch_size, num_anchors, 6].
"""
return _make.nms(data, valid_count, overlap_threshold, force_suppress, topk)
return _make.nms(data, valid_count, iou_threshold,
force_suppress, topk, id_index, do_rearrange)
Loading