Skip to content

Commit

Permalink
Add Benchmark script for PPClas models (PaddlePaddle#187)
Browse files Browse the repository at this point in the history
* first commit for yolov7

* pybind for yolov7

* CPP README.md

* CPP README.md

* modified yolov7.cc

* README.md

* python file modify

* delete license in fastdeploy/

* repush the conflict part

* README.md modified

* README.md modified

* file path modified

* file path modified

* file path modified

* file path modified

* file path modified

* README modified

* README modified

* move some helpers to private

* add examples for yolov7

* api.md modified

* api.md modified

* api.md modified

* YOLOv7

* yolov7 release link

* yolov7 release link

* yolov7 release link

* copyright

* change some helpers to private

* change variables to const and fix documents.

* gitignore

* Transfer some funtions to private member of class

* Transfer some funtions to private member of class

* Merge from develop (#9)

* Fix compile problem in different python version (#26)

* fix some usage problem in linux

* Fix compile problem

Co-authored-by: root <[email protected]>

* Add PaddleDetetion/PPYOLOE model support (#22)

* add ppdet/ppyoloe

* Add demo code and documents

* add convert processor to vision (#27)

* update .gitignore

* Added checking for cmake include dir

* fixed missing trt_backend option bug when init from trt

* remove un-need data layout and add pre-check for dtype

* changed RGB2BRG to BGR2RGB in ppcls model

* add model_zoo yolov6 c++/python demo

* fixed CMakeLists.txt typos

* update yolov6 cpp/README.md

* add yolox c++/pybind and model_zoo demo

* move some helpers to private

* fixed CMakeLists.txt typos

* add normalize with alpha and beta

* add version notes for yolov5/yolov6/yolox

* add copyright to yolov5.cc

* revert normalize

* fixed some bugs in yolox

* fixed examples/CMakeLists.txt to avoid conflicts

* add convert processor to vision

* format examples/CMakeLists summary

* Fix bug while the inference result is empty with YOLOv5 (#29)

* Add multi-label function for yolov5

* Update README.md

Update doc

* Update fastdeploy_runtime.cc

fix variable option.trt_max_shape wrong name

* Update runtime_option.md

Update resnet model dynamic shape setting name from images to x

* Fix bug when inference result boxes are empty

* Delete detection.py

Co-authored-by: Jason <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: DefTruth <[email protected]>
Co-authored-by: huangjianhui <[email protected]>

* first commit for yolor

* for merge

* Develop (#11)

* Fix compile problem in different python version (#26)

* fix some usage problem in linux

* Fix compile problem

Co-authored-by: root <[email protected]>

* Add PaddleDetetion/PPYOLOE model support (#22)

* add ppdet/ppyoloe

* Add demo code and documents

* add convert processor to vision (#27)

* update .gitignore

* Added checking for cmake include dir

* fixed missing trt_backend option bug when init from trt

* remove un-need data layout and add pre-check for dtype

* changed RGB2BRG to BGR2RGB in ppcls model

* add model_zoo yolov6 c++/python demo

* fixed CMakeLists.txt typos

* update yolov6 cpp/README.md

* add yolox c++/pybind and model_zoo demo

* move some helpers to private

* fixed CMakeLists.txt typos

* add normalize with alpha and beta

* add version notes for yolov5/yolov6/yolox

* add copyright to yolov5.cc

* revert normalize

* fixed some bugs in yolox

* fixed examples/CMakeLists.txt to avoid conflicts

* add convert processor to vision

* format examples/CMakeLists summary

* Fix bug while the inference result is empty with YOLOv5 (#29)

* Add multi-label function for yolov5

* Update README.md

Update doc

* Update fastdeploy_runtime.cc

fix variable option.trt_max_shape wrong name

* Update runtime_option.md

Update resnet model dynamic shape setting name from images to x

* Fix bug when inference result boxes are empty

* Delete detection.py

Co-authored-by: Jason <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: DefTruth <[email protected]>
Co-authored-by: huangjianhui <[email protected]>

* Yolor (#16)

* Develop (#11) (#12)

* Fix compile problem in different python version (#26)

* fix some usage problem in linux

* Fix compile problem

Co-authored-by: root <[email protected]>

* Add PaddleDetetion/PPYOLOE model support (#22)

* add ppdet/ppyoloe

* Add demo code and documents

* add convert processor to vision (#27)

* update .gitignore

* Added checking for cmake include dir

* fixed missing trt_backend option bug when init from trt

* remove un-need data layout and add pre-check for dtype

* changed RGB2BRG to BGR2RGB in ppcls model

* add model_zoo yolov6 c++/python demo

* fixed CMakeLists.txt typos

* update yolov6 cpp/README.md

* add yolox c++/pybind and model_zoo demo

* move some helpers to private

* fixed CMakeLists.txt typos

* add normalize with alpha and beta

* add version notes for yolov5/yolov6/yolox

* add copyright to yolov5.cc

* revert normalize

* fixed some bugs in yolox

* fixed examples/CMakeLists.txt to avoid conflicts

* add convert processor to vision

* format examples/CMakeLists summary

* Fix bug while the inference result is empty with YOLOv5 (#29)

* Add multi-label function for yolov5

* Update README.md

Update doc

* Update fastdeploy_runtime.cc

fix variable option.trt_max_shape wrong name

* Update runtime_option.md

Update resnet model dynamic shape setting name from images to x

* Fix bug when inference result boxes are empty

* Delete detection.py

Co-authored-by: Jason <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: DefTruth <[email protected]>
Co-authored-by: huangjianhui <[email protected]>

Co-authored-by: Jason <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: DefTruth <[email protected]>
Co-authored-by: huangjianhui <[email protected]>

* Develop (#13)

* Fix compile problem in different python version (#26)

* fix some usage problem in linux

* Fix compile problem

Co-authored-by: root <[email protected]>

* Add PaddleDetetion/PPYOLOE model support (#22)

* add ppdet/ppyoloe

* Add demo code and documents

* add convert processor to vision (#27)

* update .gitignore

* Added checking for cmake include dir

* fixed missing trt_backend option bug when init from trt

* remove un-need data layout and add pre-check for dtype

* changed RGB2BRG to BGR2RGB in ppcls model

* add model_zoo yolov6 c++/python demo

* fixed CMakeLists.txt typos

* update yolov6 cpp/README.md

* add yolox c++/pybind and model_zoo demo

* move some helpers to private

* fixed CMakeLists.txt typos

* add normalize with alpha and beta

* add version notes for yolov5/yolov6/yolox

* add copyright to yolov5.cc

* revert normalize

* fixed some bugs in yolox

* fixed examples/CMakeLists.txt to avoid conflicts

* add convert processor to vision

* format examples/CMakeLists summary

* Fix bug while the inference result is empty with YOLOv5 (#29)

* Add multi-label function for yolov5

* Update README.md

Update doc

* Update fastdeploy_runtime.cc

fix variable option.trt_max_shape wrong name

* Update runtime_option.md

Update resnet model dynamic shape setting name from images to x

* Fix bug when inference result boxes are empty

* Delete detection.py

Co-authored-by: Jason <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: DefTruth <[email protected]>
Co-authored-by: huangjianhui <[email protected]>

* documents

* documents

* documents

* documents

* documents

* documents

* documents

* documents

* documents

* documents

* documents

* documents

* Develop (#14)

* Fix compile problem in different python version (#26)

* fix some usage problem in linux

* Fix compile problem

Co-authored-by: root <[email protected]>

* Add PaddleDetetion/PPYOLOE model support (#22)

* add ppdet/ppyoloe

* Add demo code and documents

* add convert processor to vision (#27)

* update .gitignore

* Added checking for cmake include dir

* fixed missing trt_backend option bug when init from trt

* remove un-need data layout and add pre-check for dtype

* changed RGB2BRG to BGR2RGB in ppcls model

* add model_zoo yolov6 c++/python demo

* fixed CMakeLists.txt typos

* update yolov6 cpp/README.md

* add yolox c++/pybind and model_zoo demo

* move some helpers to private

* fixed CMakeLists.txt typos

* add normalize with alpha and beta

* add version notes for yolov5/yolov6/yolox

* add copyright to yolov5.cc

* revert normalize

* fixed some bugs in yolox

* fixed examples/CMakeLists.txt to avoid conflicts

* add convert processor to vision

* format examples/CMakeLists summary

* Fix bug while the inference result is empty with YOLOv5 (#29)

* Add multi-label function for yolov5

* Update README.md

Update doc

* Update fastdeploy_runtime.cc

fix variable option.trt_max_shape wrong name

* Update runtime_option.md

Update resnet model dynamic shape setting name from images to x

* Fix bug when inference result boxes are empty

* Delete detection.py

Co-authored-by: root <[email protected]>
Co-authored-by: DefTruth <[email protected]>
Co-authored-by: huangjianhui <[email protected]>

Co-authored-by: Jason <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: DefTruth <[email protected]>
Co-authored-by: huangjianhui <[email protected]>
Co-authored-by: Jason <[email protected]>

* add is_dynamic for YOLO series (#22)

* git test

* benchmark for ppclas

* retrigger ci

Co-authored-by: Jason <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: DefTruth <[email protected]>
Co-authored-by: huangjianhui <[email protected]>
Co-authored-by: Jason <[email protected]>
Co-authored-by: ziqi-jin <>
  • Loading branch information
6 people authored Sep 5, 2022
1 parent a00ef00 commit 4bf0d38
Show file tree
Hide file tree
Showing 4 changed files with 193 additions and 7 deletions.
178 changes: 178 additions & 0 deletions benchmark/vision/classification/ppclas/benchmark_ppclas_multi_input.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
import fastdeploy as fd
import cv2
import os
from tqdm import trange
import numpy as np
import datetime
import json


def parse_arguments():
import argparse
import ast
parser = argparse.ArgumentParser()
parser.add_argument(
"--model", required=True, help="Path of PaddleClas model.")
parser.add_argument(
"--image", type=str, required=False, help="Path of test image file.")
parser.add_argument(
"--input_name",
type=str,
required=False,
default="inputs",
help="input name of inference file.")
parser.add_argument(
"--topk", type=int, default=1, help="Return topk results.")
parser.add_argument(
"--cpu_num_thread",
type=int,
default=12,
help="default number of cpu thread.")
parser.add_argument(
"--size",
nargs='+',
type=int,
default=[1, 3, 224, 224],
help="size of inference array.")
parser.add_argument(
"--iter_num",
required=True,
type=int,
default=30,
help="number of iterations for computing performace.")
parser.add_argument(
"--device",
nargs='+',
type=str,
default=['cpu', 'cpu', 'cpu', 'gpu', 'gpu', 'gpu'],
help="Type of inference device, support 'cpu' or 'gpu'.")
parser.add_argument(
"--backend",
nargs='+',
type=str,
default=['ort', 'paddle', 'ov', 'ort', 'trt', 'paddle'],
help="inference backend.")
args = parser.parse_args()
backend_list = ['ov', 'trt', 'ort', 'paddle']
device_list = ['cpu', 'gpu']
assert len(args.device) == len(
args.backend), "the same number of --device and --backend is requested"
assert args.iter_num > 10, "--iter_num has to bigger than 10"
assert len(args.size
) == 4, "size should include 4 values, e.g., --size 1 3 300 300"
for b in args.backend:
assert b in backend_list, "%s backend is not supported" % b
for d in args.device:
assert d in device_list, "%s device is not supported" % d
return args


def build_option(index, args):
option = fd.RuntimeOption()
device = args.device[index]
backend = args.backend[index]
option.set_cpu_thread_num(args.cpu_num_thread)
if device == "gpu":
option.use_gpu()

if backend == "trt":
assert device == "gpu", "the trt backend need device==gpu"
option.use_trt_backend()
option.set_trt_input_shape(args.input_name, args.size)
elif backend == "ov":
assert device == "cpu", "the openvino backend need device==cpu"
option.use_openvino_backend()

elif backend == "paddle":
option.use_paddle_backend()

elif backend == "ort":
option.use_ort_backend()

else:
print("%s is an unsupported backend" % backend)

print("============= inference using %s backend on %s device ============="
% (args.backend[index], args.device[index]))
return option


args = parse_arguments()

save_dict = dict()

for index, device_name in enumerate(args.device):
if device_name not in save_dict:
save_dict[device_name] = dict()

# 配置runtime,加载模型
runtime_option = build_option(index, args)

model_file = os.path.join(args.model, "inference.pdmodel")
params_file = os.path.join(args.model, "inference.pdiparams")
config_file = os.path.join(args.model, "inference_cls.yaml")
model = fd.vision.classification.PaddleClasModel(
model_file, params_file, config_file, runtime_option=runtime_option)

# 创建要输入的向量
channel = args.size[1]
height = args.size[2]
width = args.size[3]
input_array = np.random.randint(
0, high=255, size=(height, width, channel), dtype=np.uint8)

# 如果有输入图片,则使用输入的图片进行推理
if args.image:
input_array = cv2.imread(args.image)
model_name = args.model.split('/')
model_name = model_name[-1] if model_name[-1] else model_name[-2]
print(" Model: ", model_name, " Input shape: ", input_array.shape)
start_time = datetime.datetime.now()
model.enable_record_time_of_runtime()
warmup_iter = args.iter_num // 5
warmup_end2end_time = 0
if "iter_num" not in save_dict:
save_dict["iter_num"] = args.iter_num
if "warmup_iter" not in save_dict:
save_dict["warmup_iter"] = warmup_iter
if "cpu_num_thread" not in save_dict:
save_dict["cpu_num_thread"] = args.cpu_num_thread
for i in trange(args.iter_num, desc="Inference Progress"):
if i == warmup_iter:
# 计算warmup端到端总时间(s)
warmup_time = datetime.datetime.now()
warmup_end2end_time = warmup_time - start_time
warmup_end2end_time = (
warmup_end2end_time.days * 24 * 60 * 60 +
warmup_end2end_time.seconds
) * 1000 + warmup_end2end_time.microseconds / 1000
result = model.predict(input_array, args.topk)
end_time = datetime.datetime.now()
# 计算端到端(前处理,推理,后处理)的总时间
statis_info_of_runtime_dict = model.print_statis_info_of_runtime()
end2end_time = end_time - start_time
end2end_time = (end2end_time.days * 24 * 60 * 60 + end2end_time.seconds
) * 1000 + end2end_time.microseconds / 1000
remain_end2end_time = end2end_time - warmup_end2end_time
pre_post_process = end2end_time - statis_info_of_runtime_dict[
"total_time"] * 1000
end2end = remain_end2end_time / (args.iter_num - warmup_iter)
runtime = statis_info_of_runtime_dict["avg_time"] * 1000
print("Total time of end2end: %s ms" % str(end2end_time))
print("Average time of end2end exclude warmup step: %s ms" % str(end2end))
print("Total time of preprocess and postprocess in warmup step: %s ms" %
str(warmup_end2end_time - statis_info_of_runtime_dict["warmup_time"]
* 1000))
print(
"Average time of preprocess and postprocess exclude warmup step: %s ms"
% str((remain_end2end_time - statis_info_of_runtime_dict["remain_time"]
* 1000) / (args.iter_num - warmup_iter)))
# 结构化输出
backend_name = args.backend[index]
save_dict[device_name][backend_name] = {
"end2end": end2end,
"runtime": runtime
}
json_str = json.dumps(save_dict)
with open("%s.json" % model_name, 'w', encoding='utf-8') as fw:
json.dump(json_str, fw, indent=4, ensure_ascii=False)
17 changes: 13 additions & 4 deletions csrc/fastdeploy/fastdeploy_model.cc
Original file line number Diff line number Diff line change
Expand Up @@ -162,12 +162,13 @@ bool FastDeployModel::Infer(std::vector<FDTensor>& input_tensors,
return ret;
}

void FastDeployModel::PrintStatisInfoOfRuntime() {
std::map<std::string, float> FastDeployModel::PrintStatisInfoOfRuntime() {
std::map<std::string, float> statis_info_of_runtime_dict;

if (time_of_runtime_.size() < 10) {
FDWARNING << "PrintStatisInfoOfRuntime require the runtime ran 10 times at "
"least, but now you only ran "
<< time_of_runtime_.size() << " times." << std::endl;
return;
}
double warmup_time = 0.0;
double remain_time = 0.0;
Expand All @@ -188,8 +189,16 @@ void FastDeployModel::PrintStatisInfoOfRuntime() {
std::cout << "Warmup iterations: " << warmup_iter << std::endl;
std::cout << "Total time of runtime in warmup step: " << warmup_time << "s."
<< std::endl;
std::cout << "Average time of runtime exclude warmup step: " << avg_time
<< "s." << std::endl;
std::cout << "Average time of runtime exclude warmup step: "
<< avg_time * 1000 << "ms." << std::endl;

statis_info_of_runtime_dict["total_time"] = warmup_time + remain_time;
statis_info_of_runtime_dict["warmup_time"] = warmup_time;
statis_info_of_runtime_dict["remain_time"] = remain_time;
statis_info_of_runtime_dict["warmup_iter"] = warmup_iter;
statis_info_of_runtime_dict["avg_time"] = avg_time;
statis_info_of_runtime_dict["iterations"] = time_of_runtime_.size();
return statis_info_of_runtime_dict;
}

void FastDeployModel::EnableDebug() {
Expand Down
3 changes: 1 addition & 2 deletions csrc/fastdeploy/fastdeploy_model.h
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,7 @@ class FASTDEPLOY_DECL FastDeployModel {
enable_record_time_of_runtime_ = false;
}

virtual void PrintStatisInfoOfRuntime();

virtual std::map<std::string, float> PrintStatisInfoOfRuntime();
virtual void EnableDebug();
virtual bool DebugEnabled();

Expand Down
2 changes: 1 addition & 1 deletion fastdeploy/model.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ def disable_record_time_of_runtime(self):
self._model.disable_record_time_of_runtime()

def print_statis_info_of_runtime(self):
self._model.print_statis_info_of_runtime()
return self._model.print_statis_info_of_runtime()

@property
def runtime_option(self):
Expand Down

0 comments on commit 4bf0d38

Please sign in to comment.