Added mask overlay to output image, changed fprintf info messages to …

[tehkillerbee]

• Updated object detection example with mask overlay (if supported by the model)
• Changed misc. fprintf info messages to use stdout.
• Skip detections with invalid bbox coordinates

[lvhan028]

why box.left == 0 || box.top == 0 is invalid coordinate?

[tehkillerbee]

@lvhan028 In my list of detections, I get a number of "invalid" detections with a very low score and all zero coordinates.
The bbox coordinates of these detections are usually zero and must be ignored. I figured that left/top will always be zero for these invalid detections, but after testing, it looks like valid detections can occur with a left/top coordinate of (0,0).

Maybe it would be better to look at left/top/right/bottom and make sure they are all zero. Alternatively, the score can be used to remove invalid detections.

box 318, left=47.76, top=441.28, right=85.37, bottom=476.85, label=0, score=0.3179
box 319, left=148.11, top=580.05, right=182.04, bottom=615.50, label=0, score=0.2695
box 320, left=18.68, top=470.96, right=57.28, bottom=507.18, label=0, score=0.2420
box 321, left=33.16, top=480.97, right=71.08, bottom=516.11, label=0, score=0.1003
box 322, left=439.56, top=346.23, right=571.52, bottom=480.71, label=0, score=0.0966
box 323, left=145.27, top=499.18, right=175.33, bottom=533.68, label=0, score=0.0953
box 324, left=44.71, top=325.21, right=63.88, bottom=341.79, label=0, score=0.0877
box 325, left=0.76, top=530.25, right=30.77, bottom=567.96, label=0, score=0.0854
box 326, left=244.95, top=638.06, right=295.19, bottom=680.96, label=0, score=0.0797
box 327, left=0.00, top=0.00, right=0.00, bottom=0.00, label=0, score=0.0729
box 328, left=0.00, top=0.00, right=0.00, bottom=0.00, label=0, score=0.0729
box 329, left=0.00, top=0.00, right=0.00, bottom=0.00, label=0, score=0.0729
box 330, left=0.00, top=0.00, right=0.00, bottom=0.00, label=0, score=0.0729
box 331, left=0.00, top=0.00, right=0.00, bottom=0.00, label=0, score=0.0729
box 332, left=0.00, top=0.00, right=0.00, bottom=0.00, label=0, score=0.0729
box 333, left=0.00, top=0.00, right=0.00, bottom=0.00, label=0, score=0.0729
...
box 470, left=0.00, top=0.00, right=0.00, bottom=0.00, label=0, score=0.0729
box 471, left=0.00, top=0.00, right=0.00, bottom=0.00, label=0, score=0.0729
box 472, left=0.00, top=0.00, right=0.00, bottom=0.00, label=0, score=0.0729
box 473, left=154.40, top=629.88, right=219.63, bottom=671.17, label=0, score=0.0715
box 474, left=489.83, top=569.48, right=535.17, bottom=613.19, label=0, score=0.0670
box 475, left=91.14, top=553.97, right=125.94, bottom=587.86, label=0, score=0.0656
box 476, left=583.71, top=612.17, right=644.71, bottom=707.76, label=0, score=0.0604
box 477, left=322.46, top=394.25, right=340.99, bottom=413.24, label=0, score=0.0595
box 478, left=10.53, top=503.21, right=55.76, bottom=544.70, label=0, score=0.0558
box 479, left=504.31, top=561.25, right=539.74, bottom=603.36, label=0, score=0.0521
box 480, left=146.81, top=561.98, right=181.31, bottom=595.19, label=0, score=0.0511

[lvhan028]

Thanks a lot. I think this is MMDeploy's bug. We need to investigate it.

[lzhangzz]

The current logic of the post-processing of instance segmentation is like

auto x0 = std::max(std::floor(bbox[0]) - 1, 0.f);
auto y0 = std::max(std::floor(bbox[1]) - 1, 0.f);

It is preferred to keep the same logic when recovering the whole-image mask so that the benchmark results would stay the same.

Also, the top-left corner of invalid boxes (either FP or dummy outputs from backends) is not well defined and not reliable for box filtering. In this case, edge length or area thresholding is can used in combination with score thresholding to get better visual results.

[tehkillerbee]

@lzhangzz So if I understand correctly, we should do something like:

auto x0 = std::max(std::floor(box.left) - 1, 0.f);
auto y0 = std::max(std::floor(box.top) - 1, 0.f);
cv::Rect roi((int) x0, (int) y0, mask->width, mask->height);

And for filtering the invalid boxes, we should look at not just the top-left but the side or area, in case it is zero, indicating an invalid bbox?

[lzhangzz]

auto x0 = std::max(std::floor(box.left) - 1, 0.f);
auto y0 = std::max(std::floor(box.top) - 1, 0.f);
cv::Rect roi((int) x0, (int) y0, mask->width, mask->height);

Yes

And for filtering the invalid boxes, we should look at not just the top-left but the side or area, in case it is zero, indicating an invalid bbox?

Don't use the top-left coordinate, just size/area and score.

And actually there is a "min_bbox_size" property for detection models, however it is default to 0 in most model configs. So for demostration & visualization purpose, we can perform the filtering again on the detections returned by API.

[tehkillerbee]

@lzhangzz OK, I have updated it now according to your suggestions.

[lvhan028]

LGTM

[lzhangzz]

LGTM