Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added mask overlay to output image, changed fprintf info messages to … #55

Merged
merged 4 commits into from
Jan 19, 2022
Merged

Added mask overlay to output image, changed fprintf info messages to … #55

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
cd83c71
Added mask overlay to output image, changed fprintf info messages to …
tehkillerbee Jan 7, 2022
6c5663a
Merge branch 'open-mmlab:master' into master
tehkillerbee Jan 14, 2022
c34201b
Improved box filtering (filter area/score), make sure roi coordinates…
tehkillerbee Jan 14, 2022
e83fc8d
clang-format
tehkillerbee Jan 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 25 additions & 2 deletions demo/csrc/object_detection.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,35 @@ int main(int argc, char *argv[]) {
return 1;
}

fprintf(stderr, "bbox_count=%d\n", *res_count);
fprintf(stdout, "bbox_count=%d\n", *res_count);

for (int i = 0; i < *res_count; ++i) {
const auto &box = bboxes[i].bbox;
fprintf(stderr, "box %d, left=%.2f, top=%.2f, right=%.2f, bottom=%.2f, label=%d, score=%.4f\n",
const auto &mask = bboxes[i].mask;

fprintf(stdout, "box %d, left=%.2f, top=%.2f, right=%.2f, bottom=%.2f, label=%d, score=%.4f\n",
i, box.left, box.top, box.right, box.bottom, bboxes[i].label_id, bboxes[i].score);

// skip detections with invalid bbox coordinates
if (box.left < 1 && box.top < 1) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why box.left == 0 || box.top == 0 is invalid coordinate?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lvhan028 In my list of detections, I get a number of "invalid" detections with a very low score and all zero coordinates.
The bbox coordinates of these detections are usually zero and must be ignored. I figured that left/top will always be zero for these invalid detections, but after testing, it looks like valid detections can occur with a left/top coordinate of (0,0).

Maybe it would be better to look at left/top/right/bottom and make sure they are all zero. Alternatively, the score can be used to remove invalid detections.

box 318, left=47.76, top=441.28, right=85.37, bottom=476.85, label=0, score=0.3179
box 319, left=148.11, top=580.05, right=182.04, bottom=615.50, label=0, score=0.2695
box 320, left=18.68, top=470.96, right=57.28, bottom=507.18, label=0, score=0.2420
box 321, left=33.16, top=480.97, right=71.08, bottom=516.11, label=0, score=0.1003
box 322, left=439.56, top=346.23, right=571.52, bottom=480.71, label=0, score=0.0966
box 323, left=145.27, top=499.18, right=175.33, bottom=533.68, label=0, score=0.0953
box 324, left=44.71, top=325.21, right=63.88, bottom=341.79, label=0, score=0.0877
box 325, left=0.76, top=530.25, right=30.77, bottom=567.96, label=0, score=0.0854
box 326, left=244.95, top=638.06, right=295.19, bottom=680.96, label=0, score=0.0797
box 327, left=0.00, top=0.00, right=0.00, bottom=0.00, label=0, score=0.0729
box 328, left=0.00, top=0.00, right=0.00, bottom=0.00, label=0, score=0.0729
box 329, left=0.00, top=0.00, right=0.00, bottom=0.00, label=0, score=0.0729
box 330, left=0.00, top=0.00, right=0.00, bottom=0.00, label=0, score=0.0729
box 331, left=0.00, top=0.00, right=0.00, bottom=0.00, label=0, score=0.0729
box 332, left=0.00, top=0.00, right=0.00, bottom=0.00, label=0, score=0.0729
box 333, left=0.00, top=0.00, right=0.00, bottom=0.00, label=0, score=0.0729
...
box 470, left=0.00, top=0.00, right=0.00, bottom=0.00, label=0, score=0.0729
box 471, left=0.00, top=0.00, right=0.00, bottom=0.00, label=0, score=0.0729
box 472, left=0.00, top=0.00, right=0.00, bottom=0.00, label=0, score=0.0729
box 473, left=154.40, top=629.88, right=219.63, bottom=671.17, label=0, score=0.0715
box 474, left=489.83, top=569.48, right=535.17, bottom=613.19, label=0, score=0.0670
box 475, left=91.14, top=553.97, right=125.94, bottom=587.86, label=0, score=0.0656
box 476, left=583.71, top=612.17, right=644.71, bottom=707.76, label=0, score=0.0604
box 477, left=322.46, top=394.25, right=340.99, bottom=413.24, label=0, score=0.0595
box 478, left=10.53, top=503.21, right=55.76, bottom=544.70, label=0, score=0.0558
box 479, left=504.31, top=561.25, right=539.74, bottom=603.36, label=0, score=0.0521
box 480, left=146.81, top=561.98, right=181.31, bottom=595.19, label=0, score=0.0511

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot. I think this is MMDeploy's bug. We need to investigate it.

continue;
}

// generate mask overlay if model exports masks
if (mask != nullptr) {
fprintf(stdout, "mask %d, height=%d, width=%d\n", i, mask->height, mask->width);

cv::Mat imgMask(mask->height, mask->width, CV_8UC1, &mask->data[0]);
cv::Rect roi((int)box.left - 1, (int)box.top - 1, mask->width, mask->height);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current logic of the post-processing of instance segmentation is like

auto x0 = std::max(std::floor(bbox[0]) - 1, 0.f);
auto y0 = std::max(std::floor(bbox[1]) - 1, 0.f);

It is preferred to keep the same logic when recovering the whole-image mask so that the benchmark results would stay the same.

Also, the top-left corner of invalid boxes (either FP or dummy outputs from backends) is not well defined and not reliable for box filtering. In this case, edge length or area thresholding is can used in combination with score thresholding to get better visual results.

Copy link
Contributor Author

@tehkillerbee tehkillerbee Jan 11, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lzhangzz So if I understand correctly, we should do something like:

auto x0 = std::max(std::floor(box.left) - 1, 0.f);
auto y0 = std::max(std::floor(box.top) - 1, 0.f);
cv::Rect roi((int) x0, (int) y0, mask->width, mask->height);

And for filtering the invalid boxes, we should look at not just the top-left but the side or area, in case it is zero, indicating an invalid bbox?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

auto x0 = std::max(std::floor(box.left) - 1, 0.f);
auto y0 = std::max(std::floor(box.top) - 1, 0.f);
cv::Rect roi((int) x0, (int) y0, mask->width, mask->height);

Yes

And for filtering the invalid boxes, we should look at not just the top-left but the side or area, in case it is zero, indicating an invalid bbox?

Don't use the top-left coordinate, just size/area and score.

And actually there is a "min_bbox_size" property for detection models, however it is default to 0 in most model configs. So for demostration & visualization purpose, we can perform the filtering again on the detections returned by API.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lzhangzz OK, I have updated it now according to your suggestions.


// split the RGB channels, overlay mask to a specific color channel
cv::Mat ch[3];
split(img, ch);
int col = 0; // int col = i % 3;
cv::bitwise_or(imgMask, ch[col](roi), ch[col](roi));
merge(ch, 3, img);
}

cv::rectangle(img, cv::Point{(int)box.left, (int)box.top},
cv::Point{(int)box.right, (int)box.bottom}, cv::Scalar{0, 255, 0});
}
Expand Down