Realtime performance question

[lachose1]

Hello and first of all thanks for the great tool you have built.

I wanted to ask some questions, I run the pretrained model on a 1080 ti on a realtime video and obtain so-so performances :
res50_coco_256x192: 12 FPS
mobilenetv2_coco_256x192: 15 FPS

Am I missing something or is it just that the GPU isn't strong enough. Thanks!

[innerlee]

hi, which script was run? currently the inference code is not optimized yet. see #40. you may vote here #9 to help us prioritize

[jin-s13]

Please set flip_test=False in the configs for higher speed.

[lachose1]

hi, which script was run? currently the inference code is not optimized yet. see #40. you may vote here #9 to help us prioritize

Yes, I used pretty much the same settings as #40

Please set flip_test=False in the configs for higher speed.

This was already done, but thanks for clarifying. I guess you can close the thread if this is on the road map for the future, thanks a lot for your help!

[innerlee]

thanks! it would be great if people can help with profiling and identifying the bottleneck. here is some guide we wrote earlier (can safely omit the Chinese characters and guess the content):

[fabro66]

Hi~
I found cv2.ellipse2Poly in top_dow.py/bottom_up.py highly slow the inference speed.
If I replace it with cv2.line, hrnet_w32_wholebody_256×192_dark will speed up from 3.0fps to 16.0fps on a 1060. There are other areas that can be further optimized.

Replace it with the following:

mmpose/mmpose/models/detectors/top_down.py

Line 256 in 9703521

for _, kpts in enumerate(pose_result):

    for kpts in pose_result:
        # draw each point on image
        if pose_kpt_color is not None:
            assert len(pose_kpt_color) == len(kpts)
            for kid, kpt in enumerate(kpts):
                x_coord, y_coord, kpt_score = int(kpt[0]), int(
                    kpt[1]), kpt[2]
                if kpt_score > kpt_score_thr:
                    r, g, b = pose_kpt_color[kid]
                    cv2.circle(img, (int(x_coord), int(y_coord)),
                               radius, (int(r), int(g), int(b)), -1)

        # draw limbs
        if skeleton is not None and pose_limb_color is not None:
            assert len(pose_limb_color) == len(skeleton)
            for sk_id, sk in enumerate(skeleton):
                pos1 = (int(kpts[sk[0] - 1, 0]), int(kpts[sk[0] - 1,
                                                          1]))
                pos2 = (int(kpts[sk[1] - 1, 0]), int(kpts[sk[1] - 1,
                                                          1]))
                if (pos1[0] > 0 and pos1[0] < img_w and pos1[1] > 0
                        and pos1[1] < img_h and pos2[0] > 0
                        and pos2[0] < img_w and pos2[1] > 0
                        and pos2[1] < img_h
                        and kpts[sk[0] - 1, 2] > kpt_score_thr
                        and kpts[sk[1] - 1, 2] > kpt_score_thr):

                    r, g, b = pose_limb_color[sk_id]
                    cv2.line(img, pos1, pos2, (int(r), int(g), int(b)), thickness=thickness)

[jin-s13]

Thanks for reporting @fabro66. Will try this out.

[lucasjinreal]

@fabro66 How many whole post-processing time did u measured for now?

[fabro66]

@jinfagang I did not test the time it takes for post-processing. I just replace cv2.ellipse2Poly with cv2.line to speed up inference.

[lucasjinreal]

@fabro66 Did u able to run realtime with a detector (not from GT boxes). Such as with yolov5 and a pose model.

[fabro66]

@jinfagang It can reach 16fps on a GTX1060 when I combine hrnet_w32_wholebody_256×192_dark and yolov3 (from mmdetection) to estimate whole-body keypoints.

[lucasjinreal]

@fabro66 I tested with yolov5s detector and shufflenetv2 pose on coco, the speed is about 7fps in 2 person on GTX1080ti.

What's the reason why it's slow?

[innerlee]

If you have interest, please try to profile it and post the result, something like #344 (comment)

[lucasjinreal]

@innerlee I dont know how to using cprofile. Did u guys get same performance when test a normal video with more than 2 person in it?

[lucasjinreal]

@innerlee I dont know how to using cprofile. Did u guys get same performance when test a normal video with more than 2 person in it?

[innerlee]

Step1. use cProfile to run the script for a period of time, say, 30 seconds.

Step2. visualize the result by snakeviz

Refer to the instruction in #73 (comment) for more details

[lucasjinreal]

@innerlee thanks. Do u have any insights about it?

[innerlee]

Please expand shared_transformation section, by clicking on it

[lucasjinreal]

@innerlee

[innerlee]

Please click on the shared_transform.py, and it should print more details on the bottom level.

Crop the image is not what I meant

[lucasjinreal]

@innerlee ok.....

[innerlee]

@jinfagang If you haven't deleted the profiling record, please post the result so that the bottleneck is visualized

[haseeb33]

@jinfagang Can you please share the config file for yolov5 for mmdet_model. I trained a yolov5s model independently and I want to use it as a detection model for pose estimation inference but I am having difficulties modifying the config file.
Thanks in advance.

[lucasjinreal]

@haseeb33 you can take a look at this repo: https://github.com/jinfagang/yolov7 it provides various yolo model with pure python instead yaml config files. Also support a e2e keypoints model

[haseeb33]

@jinfagang Thank you very much!