-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[TOPI][RELAY][OP] add op crop_and_resize #4417
Conversation
"bilinear - Bilinear Interpolation"); | ||
TVM_ATTR_FIELD(extrapolation_value).set_default(0.0) | ||
.describe("Specify value for extrapolation."); | ||
TVM_ATTR_FIELD(out_dtype) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is out_dtype needed? Shouldn't the out dtype = input dtype?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's actually useful to be able to specify it. Most images you'd be applying this to have datatype uint8. Bilinear or Bicubic resizing will by default output floats due to their interpolation but there may be cases where you'd like to stay as an integer type. Similarly, nearest nearest neighbor wont change the output type to float since it has no interpolation. Being able to specify a fixed output type improves the predictability of the function.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense.
TVM_DECLARE_ATTRS(CropAndResizeAttrs, "relay.attrs.CropAndResizeAttrs") { | ||
TVM_ATTR_FIELD(crop_size).set_default(NullValue<Array<IndexExpr> >()) | ||
.describe("Target Size."); | ||
TVM_ATTR_FIELD(layout).set_default("NCHW") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does it make sense to keep layout out?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Resize requires knowing which axes are the spatial dimensions, so having the layout as an attribute is necessary.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is definitely a good addition, crop_and_resize is a common op and having direct support for it will be useful. My only complaint with this implementation is the heavy duplication between resize and crop_and_resize, which I think can be dramatically reduced with a little refactoring. I recommend pulling out the kernels for nearest_neighbor, bilinear, and bicubic sampling and making them general enough to fit both types of resize.
topi/python/topi/image/resize.py
Outdated
n, c, y, x, cc = _get_indices(*indices) | ||
box_idx = box_indices(n) | ||
|
||
y1, x1 = boxes(n, 0), boxes(n, 1) | ||
y2, x2 = boxes(n, 2), boxes(n, 3) | ||
|
||
in_h = (image_height - 1) * (y2 - y1) | ||
in_w = (image_width - 1) * (x2 - x1) | ||
h_scale = tvm.div(in_h, target_h - 1) | ||
w_scale = tvm.div(in_w, target_w - 1) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree with @jwfromm This seeems to be repeated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is sligthtly different from crop, for example, h_scale is calculated based on boxes while crop has no boxes, etc. I have moved these kernels out of the op and generalized them to work for both resize and crop_and_resize.
have addressed the comments, please take a look if you have some cycles @jwfromm @anijain2305 @kevinthesun |
Looking at this again, I was wondering if there's any reason we can't implement a separate |
I guess
|
Ok, those explanations make sense. In that case I like the updates you made. LGTM once you get the tests passing. |
Overall lgtm. @yongwww Could you take a look at the ci? |
17a4479
to
02b36c1
Compare
19537ba
to
4d894b0
Compare
@kevinthesun @zhiics @jwfromm @anijain2305 fixed comments and ci. Please take another look |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Other folks please take another look as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Nice job getting this all working. I think it turned out great.
Thanks @yongwww @jwfromm @kevinthesun @anijain2305 |
* [TOPI][RELAY][OP] add op crop_and_resize * fix pylint * incorporate comments * fix ci
* [TOPI][RELAY][OP] add op crop_and_resize * fix pylint * incorporate comments * fix ci
* [TOPI][RELAY][OP] add op crop_and_resize * fix pylint * incorporate comments * fix ci
Add
crop_and_resize
op into topi and relay and TensorFlow frontend.Some of the reasons why we need
crop_and_resize
are as below:CropAndResize
is being used in TensorFlow frequently in both training and inference, 'almost' all models (such as ssd, maskrcnn, etc.) in tensorflow object detection zoo are using this op.strided_slice
,resize
andconcatenate
to composecrop_and_resize
for some cases, but the composition is not able to cover all cases, for example, whenextrapolation_value
is used,boxes
and/orbox_indices
are not list, etc.strided_slice
,resize
andconcatenate
to buildCropAndResize
will introduce a significant number of ops in the converted relay ir when there are a lot of boxes as input, it does happen in models like ssd in tensorflow object detection model zoo.@kevinthesun @zhiics @icemelon9 @jwfromm @jroesch