From 31b8648071139f7ed3bd29ab7689edcfd945a9ba Mon Sep 17 00:00:00 2001 From: Xin Chen Date: Tue, 26 Jul 2022 16:04:13 +0800 Subject: [PATCH 1/8] add prebuild package usage docs on windows --- .../02-how-to-run/prebuilt_package_windows.md | 521 ++++++++++++++++++ 1 file changed, 521 insertions(+) create mode 100644 docs/zh_cn/02-how-to-run/prebuilt_package_windows.md diff --git a/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md b/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md new file mode 100644 index 0000000000..e144a0b646 --- /dev/null +++ b/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md @@ -0,0 +1,521 @@ +# Win10 下预编译包的使用 + +- [Win10 下预编译包的使用](#win10-下预编译包的使用) + - [模型转换](#模型转换) + - [Prerequisites](#prerequisites) + - [ONNX Example](#onnx-example) + - [TensorRT Example](#tensorrt-example) + - [模型推理](#模型推理) + - [Backend Inference](#backend-inference) + - [ONNXRuntime](#onnxruntime) + - [TensorRT](#tensorrt) + - [Python SDK](#python-sdk) + - [ONNXRuntime](#onnxruntime-1) + - [TensorRT](#tensorrt-1) + - [C SDK](#c-sdk) + - [ONNXRuntime](#onnxruntime-2) + - [TensorRT](#tensorrt-2) +---- + +目前，MMDeploy 在 Windows 平台下提供 TensorRT 以及 ONNXRuntime 两种预编译包，可以从[这里](https://github.com/open-mmlab/mmdeploy/releases)获取。 本篇教程以 mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1.zip 和 mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0.zip 为例来说明预编译包的使用方法。预编译包的目录结构如下，其中dist文件夹为模型转换相关内容，sdk为模型推理相关内容。 +``` +. +|-- dist +`-- sdk + |-- bin + |-- example + |-- include + |-- lib + `-- python +``` + + +## 模型转换 + +### Prerequisites + + +**Step 0.** Download and install Miniconda from the [official website](https://docs.conda.io/en/latest/miniconda.html). + + +**Step 1.** Create a conda environment and activate it. +``` +conda create -n openmmlab python=3.8 -y +conda activate openmmlab +``` + +**Step 2.** Install PyTorch following official instructions, e.g. + +On GPU platforms: +``` +pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html +``` + +On CPU platforms: +``` +pip install torch==1.8.1+cpu torchvision==0.9.1+cpu -f https://download.pytorch.org/whl/torch_stable.html +``` + +**Step 3.** Install MMCV + +On GPU platforms: +``` +pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.8.0/index.html +``` + +On CPU platforms: +``` +pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cpu/torch1.8.0/index.html +``` + +**Step 4.** Download MMDeploy repo (just clone, no need to build) +``` +git clone https://github.com/open-mmlab/mmdeploy.git +``` + + + +### ONNX Example + +下面以Resnet分类模型来说明用法 + +**Step 0.** Install mmdeploy package +``` +pip install .\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\dist\mmdeploy-0.6.0-py38-none-win_amd64.whl +``` + +**Step 1.** Install mmclassification +``` +git clone https://github.com/open-mmlab/mmclassification.git +cd mmclassification +pip install -e . +``` + +**Step 2.** Download checkpoint from [here](https://github.com/open-mmlab/mmclassification/tree/master/configs/resnet) + +Like this [ckpt](https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_8xb32_in1k_20210831-fbbb1da6.pth) which is trained by this [config](https://github.com/open-mmlab/mmclassification/blob/master/configs/resnet/resnet18_8xb32_in1k.py) + +**Step 3.** Convert the model + +The file structure of my working directory +``` +.. +|-- mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0 +|-- mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1 +|-- mmclassification +|-- mmdeploy +`-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth +``` + +The python code to convert the model. +``` +from mmdeploy.apis import torch2onnx +from mmdeploy.backend.sdk.export_info import export2SDK + +img = 'mmclassification/demo/demo.JPEG' +work_dir = 'work_dir/onnx/resnet' +save_file = 'end2end.onnx' +deploy_cfg = 'mmdeploy/configs/mmcls/classification_onnxruntime_dynamic.py' +model_cfg = 'mmclassification/configs/resnet/resnet18_8xb32_in1k.py' +model_checkpoint = 'resnet18_8xb32_in1k_20210831-fbbb1da6.pth' +device = 'cpu' + +# 1. convert model to onnx +torch2onnx(img, work_dir, save_file, deploy_cfg, model_cfg, + model_checkpoint, device) + +# 2. extract pipeline info for sdk use (dump-info) +export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint) +``` +The file structure of work_dir after you run the python code. +``` +.\work_dir\ +`-- onnx + `-- resnet + |-- deploy.json + |-- detail.json + |-- end2end.onnx + `-- pipeline.json +``` +### TensorRT Example + +下面以Resnet分类模型来说明用法 + +**Step 0.** Install mmdeploy package +``` +pip install .\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\dist\mmdeploy-0.6.0-py38-none-win_amd64.whl +# you may add --force-reinstal if you already install mmdeploy previously +``` + +**Step 1.** Install mmclassification + +Same as ONNX Example Step 1 + +**Step 2.** Download checkpoint from [here](https://github.com/open-mmlab/mmclassification/tree/master/configs/resnet) + +Same as ONNX Example Step 2 + +**Step 3.** Install third-party package and set environment + +**Step 3.1.** Install CUDA Toolkit 11.1 + +You can download from [here](https://developer.nvidia.com/cuda-11.1.1-download-archive) + + +**Step 3.2.** Install TensorRT 8.2.3.0 + +``` +# The file structure of tensorrt package should be + +.\TensorRT-8.2.3.0\ +|-- bin +|-- data +|-- doc +|-- graphsurgeon +|-- include +|-- lib +|-- onnx_graphsurgeon +|-- python +|-- samples +`-- uff + +# Install python package +pip install .\TensorRT-8.2.3.0\python\tensorrt-8.2.3.0-cp38-none-win_amd64.whl + +# 设置环境变量 (系统属性-高级-环境变量) + +在用户的Path变量中添加TensorRT的lib路径，具体位置根据实际调整，我这里是： +C:\Deps\tensorrt\TensorRT-8.2.3.0\lib + +重启powershell让环境变量生效，可以通过 echo $env:PATH 来检查是否设置成功 + +如果你对环境变量不是很了解的话，建议只添加一个版本的TensorRT的lib到PATH里面。不建议拷贝TensorRT的dll到C盘的cuda目录，在某些情况下，这样可以暴露dll的版本问题。 +``` + +**Step 3.3.** Install cuDNN 8.2.1.0 +``` +# The file structure of cudnn package should be +|-- NVIDIA_SLA_cuDNN_Support.txt +|-- bin +|-- include +`-- lib + +# cuDNN不需要安装，只需要解压，并添加bin目录到环境变量就好，我这里是： +C:\Deps\cudnn\8.2.1\bin + +注意事项同TensorRT + +``` + +**Step 3.4.** Install pycuda +``` +pip install pycuda +``` + +**Step 4.** Convert the model + +The file structure of working directory +``` +.. +|-- mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0 +|-- mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1 +|-- mmclassification +|-- mmdeploy +`-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth +``` + +The python code to convert the model. +``` +from mmdeploy.apis import torch2onnx +from mmdeploy.apis.tensorrt import onnx2tensorrt +from mmdeploy.backend.sdk.export_info import export2SDK +import os + +img = 'mmclassification/demo/demo.JPEG' +work_dir = 'work_dir/trt/resnet' +save_file = 'end2end.onnx' +deploy_cfg = 'mmdeploy/configs/mmcls/classification_tensorrt_static-224x224.py' +model_cfg = 'mmclassification/configs/resnet/resnet18_8xb32_in1k.py' +model_checkpoint = 'resnet18_8xb32_in1k_20210831-fbbb1da6.pth' +device = 'cpu' + +# 1. convert model to IR(onnx) +torch2onnx(img, work_dir, save_file, deploy_cfg, model_cfg, + model_checkpoint, device) + +# 2. convert IR to tensorrt +onnx_model = os.path.join(work_dir, save_file) +save_file = 'end2end.engine' +model_id = 0 +device = 'cuda' +onnx2tensorrt(work_dir, save_file, model_id, deploy_cfg, onnx_model, device) + +# 3. extract pipeline info for sdk use (dump-info) +export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint) +``` + +The file structure of work_dir after you run the python code. +``` +.\work_dir\ +`-- trt + `-- resnet + |-- deploy.json + |-- detail.json + |-- end2end.engine + |-- end2end.onnx + `-- pipeline.json +``` + +## 模型推理 + +以下内容假定已完成了上述模型转换的两个Example，并得到了上述展示的两个文件夹： +``` +.\work_dir\onnx\resnet +.\work_dir\trt\resnet +``` + +### Backend Inference + +这个接口不是为了做部署的，是为了用来检验转换的模型是否可以正常推理的。 + + + +#### ONNXRuntime + +**Step 0.** Install ONNXRuntime + +**Step 0.1.** 安装onnxruntime的python包 +``` +pip install onnxruntime==1.8.1 +``` + +**Step 0.1.** 下载[onnxruntime](https://github.com/microsoft/onnxruntime/releases)，添加环境变量(这里是为了使用自定义算子) + +``` +# The file structure of onnxruntime should be +.\onnxruntime-win-gpu-x64-1.8.1\ +|-- CodeSignSummary-c0f52e3d-f27b-4c42-a587-c8479a41573c.md +|-- GIT_COMMIT_ID +|-- LICENSE +|-- Privacy.md +|-- README.md +|-- ThirdPartyNotices.txt +|-- VERSION_NUMBER +|-- include +`-- lib + +# 将lib目录添加到PATH里面，注意事项同TensorRT +``` + +**Step 1.** Inference + +Current working directory +``` +. +|-- mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0 +|-- mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1 +|-- mmclassification +|-- mmdeploy +|-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth +`-- work_dir +``` + +Python code +``` +from mmdeploy.apis import inference_model + +model_cfg = 'mmclassification/configs/resnet/resnet18_8xb32_in1k.py' +deploy_cfg = 'mmdeploy/configs/mmcls/classification_onnxruntime_dynamic.py' +backend_files = ['work_dir/onnx/resnet/end2end.onnx'] +img = 'mmclassification/demo/demo.JPEG' +device = 'cpu' +result = inference_model(model_cfg, deploy_cfg, backend_files, img, device) +``` + +**可能出现的问题：** +``` +onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Failed to load library, error code: 193 +``` +**原因：** 在较新的windows系统中，系统路径下下有两个`onnxruntime.dll`，且会优先加载，造成冲突。 +``` +C:\Windows\SysWOW64\onnxruntime.dll +C:\Windows\System32\onnxruntime.dll +``` + +**解决方法：** 以下两个方案任选其一 +1) 将系统路径下的这两个dll改名，使其加载不到，可能涉及到修改文件权限的操作 +2) 从[Github]()下载对应的版本，并将其中lib目录下的dll拷贝到mmdeploy_onnxruntime_ops.dll的同级目录 +（推荐使用Everything 进行查找，我这里是C:\Software\miniconda3\envs\openmmlab\Lib\site-packages\mmdeploy\lib\mmdeploy_onnxruntime_ops.dll） + + + +#### TensorRT + +**Step 0.** 配置环境 + +按照上述转模型的要求，安装好CUDA Toolkit 11.1，TensorRT 8.2.3.0，cuDNN 8.2.1.0 并设置好环境变量 + +**Step 1.** Inference + +Python code +``` +from mmdeploy.apis import inference_model + +model_cfg = 'mmclassification/configs/resnet/resnet18_8xb32_in1k.py' +deploy_cfg = 'mmdeploy/configs/mmcls/classification_tensorrt_static-224x224.py' +backend_files = ['work_dir/trt/resnet/end2end.engine'] +img = 'mmclassification/demo/demo.JPEG' +device = 'cuda' +result = inference_model(model_cfg, deploy_cfg, backend_files, img, device) +``` + + +### Python SDK + +这里介绍如何使用SDK的Python API进行推理 + +#### ONNXRuntime + +**Step 0.** 安装mmdeploy_python +``` +pip install .\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\python\mmdeploy_python-0.6.0-cp38-none-win_amd64.whl +``` + +**Step 1.** 下载[onnxruntime](https://github.com/microsoft/onnxruntime/releases)，添加环境变量 + + +**Step 2.** Inference +``` +python .\mmdeploy\demo\python\image_classification.py .\work_dir\onnx\resnet\ .\mmclassification\demo\demo.JPEG +``` + +#### TensorRT + +**Step 0.** 安装mmdeploy_python +``` +pip install .\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\python\mmdeploy_python-0.6.0-cp38-none-win_amd64.whl +``` + +**Step 1.** 按照上述转模型的要求，安装好CUDA Toolkit 11.1，TensorRT 8.2.3.0，cuDNN 8.2.1.0 并设置好环境变量 + + +**Step 2.** Inference +``` + python .\mmdeploy\demo\python\image_classification.py .\work_dir\trt\resnet\ .\mmclassification\demo\demo.JPEG --device-name cuda +``` + + +### C SDK + +这里介绍如何使用SDK的C API进行推理 + +这里涉及到编译，需要安装Vs2019+，CMake + +example中读取图片用到了OpenCV，所以需要从这里安装[opencv-4.6.0-vc14_vc15.exe](https://github.com/opencv/opencv/releases)，或者自行编译 + + +``` +// Current working directorys +. +|-- opencv +|-- mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0 +|-- mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1 +|-- mmclassification +|-- mmdeploy +|-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth +`-- work_dir +``` + +#### ONNXRuntime + +**Step 0.** 编译 + +在mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\example目录下 +``` +// 部分路径根据所在硬盘的位置进行修改 +mkdir build +cd build +cmake .. -A x64 -T v142 ` + -DOpenCV_DIR=C:\workspace\opencv\build\x64\vc15\lib ` + -DMMDeploy_DIR=C:\workspace\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\lib\cmake\MMDeploy ` + -DONNXRUNTIME_DIR=C:\Deps\onnxruntime\onnxruntime-win-gpu-x64-1.8.1 + +cmake --build . --config Release + +``` + +**Step 1.** 添加环境变量 + +需要添加的环境变量有三个：分别是OpenCV的bin目录，mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\bin，以及onnxruntime的lib目录。在之前已经添加过的变量可以忽略，我这里是： +``` +C:\Deps\onnxruntime\onnxruntime-win-gpu-x64-1.8.1\lib +C:\workspace\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\bin +C:\workspace\opencv\build\x64\vc15\bin +``` + +这里也可以不添加环境变量，而将这三者的dll拷贝到刚才编译出的exe(build/Release)的同级目录下。 + +**Step 2.** Inference +重启Powershell让环境变量生效。这里建议使用cmd，这样如果exe运行时如果找不到相关的dll的话会有弹窗 +在mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\example\build\Release目录下： +``` +.\image_classification.exe cpu C:\workspace\work_dir\onnx\resnet\ C:\workspace\mmclassification\demo\demo.JPEG +``` + + +#### TensorRT + +**Step 0.** 编译 + +在mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\example目录下 +``` +// 部分路径根据所在硬盘的位置进行修改 +mkdir build +cd build +cmake .. -A x64 -T v142 ` + -DOpenCV_DIR=C:\workspace\opencv\build\x64\vc15\lib ` + -DMMDeploy_DIR=C:\workspace\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\lib\cmake\MMDeploy ` + -DTENSORRT_DIR=C:\Deps\tensorrt\TensorRT-8.2.3.0 ` + -DCUDNN_DIR=C:\Deps\cudnn\8.2.1 + +cmake --build . --config Release +``` + +**可能出现的问题：** +``` +enable_language(CUDA) 报错 + +-- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.19044. +-- Found CUDA: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.1 (found version "11.1") +CMake Error at C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCompilerId.cmake:491 (message): + No CUDA toolset found. +Call Stack (most recent call first): + C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCompilerId.cmake:6 (CMAKE_DETERMINE_COMPILER_ID_BUILD) + C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCompilerId.cmake:59 (__determine_compiler_id_test) + C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCUDACompiler.cmake:339 (CMAKE_DETERMINE_COMPILER_ID) + C:/workspace/mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0/sdk/lib/cmake/MMDeploy/MMDeployConfig.cmake:27 (enable_language) + CMakeLists.txt:5 (find_package) +``` + +**原因：** CUDA Toolkit 11.1安装在Visual Studio之前，造成VS的插件没有安装。或者VS的版本过新，使得CUDA Toolkit的安装的时候跳过了VS插件的安装 + + +**解决方法：** 可以通过手工拷贝插件的方式来解决这个问题。 +我这里的环境是CUDA Toolkit 11.1，vs2022，操作是将`C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\extras\visual_studio_integration\MSBuildExtensions`中的四个文件拷贝到`C:\Software\Microsoft Visual Studio\2022\Community\Msbuild\Microsoft\VC\v170\BuildCustomizations` 目录下。具体路径根据实际情况进行更改。 + + +**Step 1.** 添加环境变量 +这里需要添加以下四个变量，根据各自的情况进行调整 +``` +C:\Deps\cudnn\8.2.1\bin +C:\Deps\tensorrt\TensorRT-8.2.3.0\lib +C:\workspace\opencv\build\x64\vc15\bin +C:\workspace\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\bin +``` + +**Step 2.** Inference +重启Powershell让环境变量生效。这里建议使用cmd，这样如果exe运行时如果找不到相关的dll的话会有弹窗 +在mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\example\build\Release目录下： +``` +.\image_classification.exe cuda C:\workspace\work_dir\trt\resnet C:\workspace\mmclassification\demo\demo.JPEG +``` From ea19e125d6b8163c910706bd1652b9ad1c042780 Mon Sep 17 00:00:00 2001 From: Xin Chen Date: Tue, 26 Jul 2022 16:09:56 +0800 Subject: [PATCH 2/8] fix lint --- .../02-how-to-run/prebuilt_package_windows.md | 89 ++++++++++++------- 1 file changed, 56 insertions(+), 33 deletions(-) diff --git a/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md b/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md index e144a0b646..417679864f 100644 --- a/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md +++ b/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md @@ -15,9 +15,11 @@ - [C SDK](#c-sdk) - [ONNXRuntime](#onnxruntime-2) - [TensorRT](#tensorrt-2) ----- + +______________________________________________________________________ 目前，MMDeploy 在 Windows 平台下提供 TensorRT 以及 ONNXRuntime 两种预编译包，可以从[这里](https://github.com/open-mmlab/mmdeploy/releases)获取。 本篇教程以 mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1.zip 和 mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0.zip 为例来说明预编译包的使用方法。预编译包的目录结构如下，其中dist文件夹为模型转换相关内容，sdk为模型推理相关内容。 + ``` . |-- dist @@ -29,29 +31,29 @@ `-- python ``` - ## 模型转换 ### Prerequisites - **Step 0.** Download and install Miniconda from the [official website](https://docs.conda.io/en/latest/miniconda.html). - **Step 1.** Create a conda environment and activate it. + ``` conda create -n openmmlab python=3.8 -y -conda activate openmmlab +conda activate openmmlab ``` **Step 2.** Install PyTorch following official instructions, e.g. On GPU platforms: + ``` pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html ``` On CPU platforms: + ``` pip install torch==1.8.1+cpu torchvision==0.9.1+cpu -f https://download.pytorch.org/whl/torch_stable.html ``` @@ -59,32 +61,35 @@ pip install torch==1.8.1+cpu torchvision==0.9.1+cpu -f https://download.pytorch. **Step 3.** Install MMCV On GPU platforms: + ``` pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.8.0/index.html ``` On CPU platforms: + ``` pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cpu/torch1.8.0/index.html ``` **Step 4.** Download MMDeploy repo (just clone, no need to build) + ``` git clone https://github.com/open-mmlab/mmdeploy.git ``` - - ### ONNX Example 下面以Resnet分类模型来说明用法 **Step 0.** Install mmdeploy package + ``` pip install .\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\dist\mmdeploy-0.6.0-py38-none-win_amd64.whl ``` **Step 1.** Install mmclassification + ``` git clone https://github.com/open-mmlab/mmclassification.git cd mmclassification @@ -98,6 +103,7 @@ Like this [ckpt](https://download.openmmlab.com/mmclassification/v0/resnet/resne **Step 3.** Convert the model The file structure of my working directory + ``` .. |-- mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0 @@ -108,6 +114,7 @@ The file structure of my working directory ``` The python code to convert the model. + ``` from mmdeploy.apis import torch2onnx from mmdeploy.backend.sdk.export_info import export2SDK @@ -127,7 +134,9 @@ torch2onnx(img, work_dir, save_file, deploy_cfg, model_cfg, # 2. extract pipeline info for sdk use (dump-info) export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint) ``` + The file structure of work_dir after you run the python code. + ``` .\work_dir\ `-- onnx @@ -137,11 +146,13 @@ The file structure of work_dir after you run the python code. |-- end2end.onnx `-- pipeline.json ``` + ### TensorRT Example 下面以Resnet分类模型来说明用法 **Step 0.** Install mmdeploy package + ``` pip install .\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\dist\mmdeploy-0.6.0-py38-none-win_amd64.whl # you may add --force-reinstal if you already install mmdeploy previously @@ -161,7 +172,6 @@ Same as ONNX Example Step 2 You can download from [here](https://developer.nvidia.com/cuda-11.1.1-download-archive) - **Step 3.2.** Install TensorRT 8.2.3.0 ``` @@ -193,6 +203,7 @@ C:\Deps\tensorrt\TensorRT-8.2.3.0\lib ``` **Step 3.3.** Install cuDNN 8.2.1.0 + ``` # The file structure of cudnn package should be |-- NVIDIA_SLA_cuDNN_Support.txt @@ -208,6 +219,7 @@ C:\Deps\cudnn\8.2.1\bin ``` **Step 3.4.** Install pycuda + ``` pip install pycuda ``` @@ -215,6 +227,7 @@ pip install pycuda **Step 4.** Convert the model The file structure of working directory + ``` .. |-- mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0 @@ -225,6 +238,7 @@ The file structure of working directory ``` The python code to convert the model. + ``` from mmdeploy.apis import torch2onnx from mmdeploy.apis.tensorrt import onnx2tensorrt @@ -255,6 +269,7 @@ export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint) ``` The file structure of work_dir after you run the python code. + ``` .\work_dir\ `-- trt @@ -269,6 +284,7 @@ The file structure of work_dir after you run the python code. ## 模型推理 以下内容假定已完成了上述模型转换的两个Example，并得到了上述展示的两个文件夹： + ``` .\work_dir\onnx\resnet .\work_dir\trt\resnet @@ -276,15 +292,14 @@ The file structure of work_dir after you run the python code. ### Backend Inference -这个接口不是为了做部署的，是为了用来检验转换的模型是否可以正常推理的。 - - +这个接口不是为了做部署的，是为了用来检验转换的模型是否可以正常推理的。 #### ONNXRuntime **Step 0.** Install ONNXRuntime **Step 0.1.** 安装onnxruntime的python包 + ``` pip install onnxruntime==1.8.1 ``` @@ -310,6 +325,7 @@ pip install onnxruntime==1.8.1 **Step 1.** Inference Current working directory + ``` . |-- mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0 @@ -321,6 +337,7 @@ Current working directory ``` Python code + ``` from mmdeploy.apis import inference_model @@ -333,23 +350,25 @@ result = inference_model(model_cfg, deploy_cfg, backend_files, img, device) ``` **可能出现的问题：** + ``` onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Failed to load library, error code: 193 ``` + **原因：** 在较新的windows系统中，系统路径下下有两个`onnxruntime.dll`，且会优先加载，造成冲突。 + ``` C:\Windows\SysWOW64\onnxruntime.dll C:\Windows\System32\onnxruntime.dll ``` -**解决方法：** 以下两个方案任选其一 -1) 将系统路径下的这两个dll改名，使其加载不到，可能涉及到修改文件权限的操作 -2) 从[Github]()下载对应的版本，并将其中lib目录下的dll拷贝到mmdeploy_onnxruntime_ops.dll的同级目录 -（推荐使用Everything 进行查找，我这里是C:\Software\miniconda3\envs\openmmlab\Lib\site-packages\mmdeploy\lib\mmdeploy_onnxruntime_ops.dll） +**解决方法：** 以下两个方案任选其一 +1. 将系统路径下的这两个dll改名，使其加载不到，可能涉及到修改文件权限的操作 +2. 从[Github](<>)下载对应的版本，并将其中lib目录下的dll拷贝到mmdeploy_onnxruntime_ops.dll的同级目录 + （推荐使用Everything 进行查找，我这里是C:\\Software\\miniconda3\\envs\\openmmlab\\Lib\\site-packages\\mmdeploy\\lib\\mmdeploy_onnxruntime_ops.dll） - -#### TensorRT +#### TensorRT **Step 0.** 配置环境 @@ -358,6 +377,7 @@ C:\Windows\System32\onnxruntime.dll **Step 1.** Inference Python code + ``` from mmdeploy.apis import inference_model @@ -369,7 +389,6 @@ device = 'cuda' result = inference_model(model_cfg, deploy_cfg, backend_files, img, device) ``` - ### Python SDK 这里介绍如何使用SDK的Python API进行推理 @@ -377,34 +396,35 @@ result = inference_model(model_cfg, deploy_cfg, backend_files, img, device) #### ONNXRuntime **Step 0.** 安装mmdeploy_python + ``` pip install .\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\python\mmdeploy_python-0.6.0-cp38-none-win_amd64.whl ``` **Step 1.** 下载[onnxruntime](https://github.com/microsoft/onnxruntime/releases)，添加环境变量 - **Step 2.** Inference + ``` python .\mmdeploy\demo\python\image_classification.py .\work_dir\onnx\resnet\ .\mmclassification\demo\demo.JPEG ``` -#### TensorRT +#### TensorRT **Step 0.** 安装mmdeploy_python + ``` pip install .\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\python\mmdeploy_python-0.6.0-cp38-none-win_amd64.whl ``` **Step 1.** 按照上述转模型的要求，安装好CUDA Toolkit 11.1，TensorRT 8.2.3.0，cuDNN 8.2.1.0 并设置好环境变量 - **Step 2.** Inference + ``` python .\mmdeploy\demo\python\image_classification.py .\work_dir\trt\resnet\ .\mmclassification\demo\demo.JPEG --device-name cuda ``` - ### C SDK 这里介绍如何使用SDK的C API进行推理 @@ -413,9 +433,8 @@ pip install .\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\python\m example中读取图片用到了OpenCV，所以需要从这里安装[opencv-4.6.0-vc14_vc15.exe](https://github.com/opencv/opencv/releases)，或者自行编译 - ``` -// Current working directorys +// Current working directories . |-- opencv |-- mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0 @@ -430,7 +449,8 @@ example中读取图片用到了OpenCV，所以需要从这里安装[opencv-4.6.0 **Step 0.** 编译 -在mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\example目录下 +在mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\\sdk\\example目录下 + ``` // 部分路径根据所在硬盘的位置进行修改 mkdir build @@ -446,7 +466,8 @@ cmake --build . --config Release **Step 1.** 添加环境变量 -需要添加的环境变量有三个：分别是OpenCV的bin目录，mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\bin，以及onnxruntime的lib目录。在之前已经添加过的变量可以忽略，我这里是： +需要添加的环境变量有三个：分别是OpenCV的bin目录，mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\\sdk\\bin，以及onnxruntime的lib目录。在之前已经添加过的变量可以忽略，我这里是： + ``` C:\Deps\onnxruntime\onnxruntime-win-gpu-x64-1.8.1\lib C:\workspace\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\bin @@ -457,17 +478,18 @@ C:\workspace\opencv\build\x64\vc15\bin **Step 2.** Inference 重启Powershell让环境变量生效。这里建议使用cmd，这样如果exe运行时如果找不到相关的dll的话会有弹窗 -在mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\example\build\Release目录下： +在mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\\sdk\\example\\build\\Release目录下： + ``` .\image_classification.exe cpu C:\workspace\work_dir\onnx\resnet\ C:\workspace\mmclassification\demo\demo.JPEG ``` - -#### TensorRT +#### TensorRT **Step 0.** 编译 -在mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\example目录下 +在mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\\sdk\\example目录下 + ``` // 部分路径根据所在硬盘的位置进行修改 mkdir build @@ -482,6 +504,7 @@ cmake --build . --config Release ``` **可能出现的问题：** + ``` enable_language(CUDA) 报错 @@ -499,13 +522,12 @@ Call Stack (most recent call first): **原因：** CUDA Toolkit 11.1安装在Visual Studio之前，造成VS的插件没有安装。或者VS的版本过新，使得CUDA Toolkit的安装的时候跳过了VS插件的安装 - **解决方法：** 可以通过手工拷贝插件的方式来解决这个问题。 我这里的环境是CUDA Toolkit 11.1，vs2022，操作是将`C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\extras\visual_studio_integration\MSBuildExtensions`中的四个文件拷贝到`C:\Software\Microsoft Visual Studio\2022\Community\Msbuild\Microsoft\VC\v170\BuildCustomizations` 目录下。具体路径根据实际情况进行更改。 - **Step 1.** 添加环境变量 这里需要添加以下四个变量，根据各自的情况进行调整 + ``` C:\Deps\cudnn\8.2.1\bin C:\Deps\tensorrt\TensorRT-8.2.3.0\lib @@ -515,7 +537,8 @@ C:\workspace\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\bin **Step 2.** Inference 重启Powershell让环境变量生效。这里建议使用cmd，这样如果exe运行时如果找不到相关的dll的话会有弹窗 -在mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\example\build\Release目录下： +在mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\\sdk\\example\\build\\Release目录下： + ``` .\image_classification.exe cuda C:\workspace\work_dir\trt\resnet C:\workspace\mmclassification\demo\demo.JPEG ``` From 0256dcb8aa058fdb780cc55b45758008660b3a2d Mon Sep 17 00:00:00 2001 From: Xin Chen Date: Thu, 28 Jul 2022 20:28:28 +0800 Subject: [PATCH 3/8] update --- .../02-how-to-run/prebuilt_package_windows.md | 514 ++++++------------ docs/zh_cn/faq.md | 33 ++ 2 files changed, 195 insertions(+), 352 deletions(-) diff --git a/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md b/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md index 417679864f..4b0027d020 100644 --- a/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md +++ b/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md @@ -1,25 +1,33 @@ # Win10 下预编译包的使用 - [Win10 下预编译包的使用](#win10-下预编译包的使用) + - [- 可能遇到的问题](#--可能遇到的问题) + - [准备工作](#准备工作) + - [ONNX Runtime](#onnx-runtime) + - [TensorRT](#tensorrt) - [模型转换](#模型转换) - - [Prerequisites](#prerequisites) - - [ONNX Example](#onnx-example) + - [ONNX Runtime Example](#onnx-runtime-example) - [TensorRT Example](#tensorrt-example) - [模型推理](#模型推理) - [Backend Inference](#backend-inference) - [ONNXRuntime](#onnxruntime) - - [TensorRT](#tensorrt) + - [TensorRT](#tensorrt-1) - [Python SDK](#python-sdk) - [ONNXRuntime](#onnxruntime-1) - - [TensorRT](#tensorrt-1) + - [TensorRT](#tensorrt-2) - [C SDK](#c-sdk) - [ONNXRuntime](#onnxruntime-2) - - [TensorRT](#tensorrt-2) + - [TensorRT](#tensorrt-3) + - [可能遇到的问题](#可能遇到的问题) +---- + +目前，`MMDeploy`在`Windows`平台下提供`TensorRT`以及`ONNX Runtime`两种预编译包，可以从[Releases](https://github.com/open-mmlab/mmdeploy/releases)获取。 -______________________________________________________________________ +本篇教程以`mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1.zip`和`mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0.zip`为例，展示预编译包的使用方法。 -目前，MMDeploy 在 Windows 平台下提供 TensorRT 以及 ONNXRuntime 两种预编译包，可以从[这里](https://github.com/open-mmlab/mmdeploy/releases)获取。 本篇教程以 mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1.zip 和 mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0.zip 为例来说明预编译包的使用方法。预编译包的目录结构如下，其中dist文件夹为模型转换相关内容，sdk为模型推理相关内容。 +为了方便使用者快速上手，本教程以分类模型(mmclassification)为例，展示两种预编译包的使用方法。 +预编译包的目录结构如下，其中`dist`文件夹为模型转换相关内容，`sdk`文件夹为模型推理相关内容。 ``` . |-- dist @@ -31,91 +39,103 @@ ______________________________________________________________________ `-- python ``` -## 模型转换 - -### Prerequisites - -**Step 0.** Download and install Miniconda from the [official website](https://docs.conda.io/en/latest/miniconda.html). +## 准备工作 -**Step 1.** Create a conda environment and activate it. +使用预编译包来进行`模型转换`以及`模型推理`，除了预编译包的中的内容外，还需要安装一些第三方依赖库，下面分别介绍以`ONNX Runtime`、`TensorRT`为推理后端所要进行的准备工作。 -``` -conda create -n openmmlab python=3.8 -y -conda activate openmmlab -``` +两种推理后端环境准备工作中，其中一些操作是共有的，下面先介绍这些共有的操作，再分别介绍各自特有的操作。 -**Step 2.** Install PyTorch following official instructions, e.g. +首先新建一个工作目录workspace -On GPU platforms: +1. 请按照[get_started](../get_started.md)文档，准备虚拟环境，安装pytorch、torchvision、mmcv-full。若要使用SDK的C接口，需要安装vs2019+, OpenCV。 + :point_right: 这里建议使用`pip`而不是`conda`安装pytorch、torchvision +2. 克隆mmdeploy仓库 + ``` bash + git clone https://github.com/open-mmlab/mmdeploy.git + ``` + :point_right: 这里主要为了使用configs文件，所以没有加`--recursive`来下载submodule，也不需要编译`mmdeploy` +3. 安装mmclassification + ``` bash + git clone https://github.com/open-mmlab/mmclassification.git + cd mmclassification + pip install -e . + ``` +4. 准备一个PyTorch的模型文件当作我们的示例 + 这里选择了[resnet18_8xb32_in1k_20210831-fbbb1da6.pth](https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_8xb32_in1k_20210831-fbbb1da6.pth)，对应的训练config为[resnet18_8xb32_in1k.py](https://github.com/open-mmlab/mmclassification/blob/master/configs/resnet/resnet18_8xb32_in1k.py) +做好以上工作后，当前工作目录的结构应为： ``` -pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html -``` - -On CPU platforms: - -``` -pip install torch==1.8.1+cpu torchvision==0.9.1+cpu -f https://download.pytorch.org/whl/torch_stable.html -``` - -**Step 3.** Install MMCV - -On GPU platforms: - -``` -pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.8.0/index.html +. +|-- mmclassification +|-- mmdeploy +|-- opencv +|-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth ``` -On CPU platforms: -``` -pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cpu/torch1.8.0/index.html -``` -**Step 4.** Download MMDeploy repo (just clone, no need to build) +### ONNX Runtime -``` -git clone https://github.com/open-mmlab/mmdeploy.git -``` +本节介绍`mmdeploy`使用`ONNX Runtime`推理所特有的环境准备工作 -### ONNX Example +5. 安装`mmdeploy`（模型转换）以及`mmdeploy_python`（模型推理Python API）的预编译包 + ``` bash + # 先下载 mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1.zip + pip install .\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\dist\mmdeploy-0.6.0-py38-none-win_amd64.whl + pip install .\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\python\mmdeploy_python-0.6.0-cp38-none-win_amd64.whl + ``` + :point_right: 如果之前安装过，需要先卸载后再安装。 +6. 安装onnxruntime package + ``` + pip install onnxruntime==1.8.1 + ``` +7. 下载[`onnxruntime`](https://github.com/microsoft/onnxruntime/releases/tag/v1.8.1)，添加环境变量 + 将onnxruntime的lib目录添加到PATH里面，如图所示，具体的路径根据个人情况更改。 + + ![sys-path](https://user-images.githubusercontent.com/16019484/181463801-1d7814a8-b256-46e9-86f2-c08de0bc150b.png) + :exclamation: 重启powershell让环境变量生效，可以通过 echo $env:PATH 来检查是否设置成功。 -下面以Resnet分类模型来说明用法 -**Step 0.** Install mmdeploy package +### TensorRT -``` -pip install .\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\dist\mmdeploy-0.6.0-py38-none-win_amd64.whl -``` +本节介绍`mmdeploy`使用`TensorRT`推理所特有的环境准备工作 -**Step 1.** Install mmclassification +5. 安装`mmdeploy`（模型转换）以及`mmdeploy_python`（模型推理Python API）的预编译包 + ``` bash + # 先下载 mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0.zip + pip install .\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\dist\mmdeploy-0.6.0-py38-none-win_amd64.whl + pip install .\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\python\mmdeploy_python-0.6.0-cp38-none-win_amd64.whl + ``` + :point_right: 如果之前安装过，需要先卸载后再安装 +6. 安装CUDA相关内容，并设置环境变量 + - CUDA Toolkit 11.1 + - TensorRT 8.2.3.0 (python包 + 环境变量) + - cuDNN 8.2.1.0 + + 其中CUDA的环境变量在安装CUDA Toolkit后会自动添加，TensorRT以及cuDNN解压后需要自行添加运行库的路径到PATH，可参考onnxruntime的设置图例 + :exclamation: 重启powershell让环境变量生效，可以通过 echo $env:PATH 来检查是否设置成功 + :exclamation: 建议只添加一个版本的TensorRT的lib到PATH里面。不建议拷贝TensorRT的dll到C盘的cuda目录，在某些情况下，这样可以暴露dll的版本问题 +7. 安装pycuda `pip install pycuda` -``` -git clone https://github.com/open-mmlab/mmclassification.git -cd mmclassification -pip install -e . -``` - -**Step 2.** Download checkpoint from [here](https://github.com/open-mmlab/mmclassification/tree/master/configs/resnet) +## 模型转换 -Like this [ckpt](https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_8xb32_in1k_20210831-fbbb1da6.pth) which is trained by this [config](https://github.com/open-mmlab/mmclassification/blob/master/configs/resnet/resnet18_8xb32_in1k.py) -**Step 3.** Convert the model +### ONNX Runtime Example -The file structure of my working directory +下面介绍根据之前下载的ckpt来展示如果使用`mmdeploy`预编译包来进行模型转换 +经过之前的准备工作，当前的工作目录结构应该为： ``` .. -|-- mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0 |-- mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1 |-- mmclassification |-- mmdeploy +|-- opencv `-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth ``` -The python code to convert the model. - -``` +python 转换代码 +```python from mmdeploy.apis import torch2onnx from mmdeploy.backend.sdk.export_info import export2SDK @@ -135,9 +155,8 @@ torch2onnx(img, work_dir, save_file, deploy_cfg, model_cfg, export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint) ``` -The file structure of work_dir after you run the python code. - -``` +转换后的模型目录结构应该为： +``` bash .\work_dir\ `-- onnx `-- resnet @@ -146,100 +165,23 @@ The file structure of work_dir after you run the python code. |-- end2end.onnx `-- pipeline.json ``` - ### TensorRT Example -下面以Resnet分类模型来说明用法 - -**Step 0.** Install mmdeploy package - -``` -pip install .\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\dist\mmdeploy-0.6.0-py38-none-win_amd64.whl -# you may add --force-reinstal if you already install mmdeploy previously -``` - -**Step 1.** Install mmclassification - -Same as ONNX Example Step 1 - -**Step 2.** Download checkpoint from [here](https://github.com/open-mmlab/mmclassification/tree/master/configs/resnet) - -Same as ONNX Example Step 2 - -**Step 3.** Install third-party package and set environment - -**Step 3.1.** Install CUDA Toolkit 11.1 +下面根据之前下载的ckpt来展示如果使用mmdeploy预编译包来进行模型转换 -You can download from [here](https://developer.nvidia.com/cuda-11.1.1-download-archive) - -**Step 3.2.** Install TensorRT 8.2.3.0 - -``` -# The file structure of tensorrt package should be - -.\TensorRT-8.2.3.0\ -|-- bin -|-- data -|-- doc -|-- graphsurgeon -|-- include -|-- lib -|-- onnx_graphsurgeon -|-- python -|-- samples -`-- uff - -# Install python package -pip install .\TensorRT-8.2.3.0\python\tensorrt-8.2.3.0-cp38-none-win_amd64.whl - -# 设置环境变量 (系统属性-高级-环境变量) - -在用户的Path变量中添加TensorRT的lib路径，具体位置根据实际调整，我这里是： -C:\Deps\tensorrt\TensorRT-8.2.3.0\lib - -重启powershell让环境变量生效，可以通过 echo $env:PATH 来检查是否设置成功 - -如果你对环境变量不是很了解的话，建议只添加一个版本的TensorRT的lib到PATH里面。不建议拷贝TensorRT的dll到C盘的cuda目录，在某些情况下，这样可以暴露dll的版本问题。 -``` - -**Step 3.3.** Install cuDNN 8.2.1.0 - -``` -# The file structure of cudnn package should be -|-- NVIDIA_SLA_cuDNN_Support.txt -|-- bin -|-- include -`-- lib - -# cuDNN不需要安装，只需要解压，并添加bin目录到环境变量就好，我这里是： -C:\Deps\cudnn\8.2.1\bin - -注意事项同TensorRT - -``` - -**Step 3.4.** Install pycuda - -``` -pip install pycuda -``` - -**Step 4.** Convert the model - -The file structure of working directory +经过之前的准备工作，当前的工作目录结构应该为： ``` .. |-- mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0 -|-- mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1 |-- mmclassification |-- mmdeploy +|-- opencv `-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth ``` -The python code to convert the model. - -``` +python 转换代码 +``` python from mmdeploy.apis import torch2onnx from mmdeploy.apis.tensorrt import onnx2tensorrt from mmdeploy.backend.sdk.export_info import export2SDK @@ -268,8 +210,7 @@ onnx2tensorrt(work_dir, save_file, model_id, deploy_cfg, onnx_model, device) export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint) ``` -The file structure of work_dir after you run the python code. - +转换后的模型目录结构应该为： ``` .\work_dir\ `-- trt @@ -283,62 +224,32 @@ The file structure of work_dir after you run the python code. ## 模型推理 -以下内容假定已完成了上述模型转换的两个Example，并得到了上述展示的两个文件夹： - +以下内容假定已完成了上述模型转换的两个Example，并得到了上述模型转换后的两个文件夹其中之一或者全部： ``` .\work_dir\onnx\resnet .\work_dir\trt\resnet ``` -### Backend Inference - -这个接口不是为了做部署的，是为了用来检验转换的模型是否可以正常推理的。 - -#### ONNXRuntime - -**Step 0.** Install ONNXRuntime - -**Step 0.1.** 安装onnxruntime的python包 - -``` -pip install onnxruntime==1.8.1 -``` - -**Step 0.1.** 下载[onnxruntime](https://github.com/microsoft/onnxruntime/releases)，添加环境变量(这里是为了使用自定义算子) - -``` -# The file structure of onnxruntime should be -.\onnxruntime-win-gpu-x64-1.8.1\ -|-- CodeSignSummary-c0f52e3d-f27b-4c42-a587-c8479a41573c.md -|-- GIT_COMMIT_ID -|-- LICENSE -|-- Privacy.md -|-- README.md -|-- ThirdPartyNotices.txt -|-- VERSION_NUMBER -|-- include -`-- lib - -# 将lib目录添加到PATH里面，注意事项同TensorRT -``` - -**Step 1.** Inference - -Current working directory - +当前的工作目录应为： ``` . |-- mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0 |-- mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1 |-- mmclassification |-- mmdeploy +|-- opencv |-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth `-- work_dir ``` -Python code +### Backend Inference -``` +:exclamation: 需要强调的一点是，这个接口不是为了做部署的，而是屏蔽了推理后端接口的，用来检验转换的模型是否可以正常推理的。 + +#### ONNXRuntime + +Python 代码 +``` python from mmdeploy.apis import inference_model model_cfg = 'mmclassification/configs/resnet/resnet18_8xb32_in1k.py' @@ -349,36 +260,10 @@ device = 'cpu' result = inference_model(model_cfg, deploy_cfg, backend_files, img, device) ``` -**可能出现的问题：** - -``` -onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Failed to load library, error code: 193 -``` - -**原因：** 在较新的windows系统中，系统路径下下有两个`onnxruntime.dll`，且会优先加载，造成冲突。 - -``` -C:\Windows\SysWOW64\onnxruntime.dll -C:\Windows\System32\onnxruntime.dll -``` - -**解决方法：** 以下两个方案任选其一 - -1. 将系统路径下的这两个dll改名，使其加载不到，可能涉及到修改文件权限的操作 -2. 从[Github](<>)下载对应的版本，并将其中lib目录下的dll拷贝到mmdeploy_onnxruntime_ops.dll的同级目录 - （推荐使用Everything 进行查找，我这里是C:\\Software\\miniconda3\\envs\\openmmlab\\Lib\\site-packages\\mmdeploy\\lib\\mmdeploy_onnxruntime_ops.dll） - -#### TensorRT - -**Step 0.** 配置环境 +#### TensorRT -按照上述转模型的要求，安装好CUDA Toolkit 11.1，TensorRT 8.2.3.0，cuDNN 8.2.1.0 并设置好环境变量 - -**Step 1.** Inference - -Python code - -``` +Python 代码 +``` python from mmdeploy.apis import inference_model model_cfg = 'mmclassification/configs/resnet/resnet18_8xb32_in1k.py' @@ -395,32 +280,14 @@ result = inference_model(model_cfg, deploy_cfg, backend_files, img, device) #### ONNXRuntime -**Step 0.** 安装mmdeploy_python - -``` -pip install .\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\python\mmdeploy_python-0.6.0-cp38-none-win_amd64.whl -``` - -**Step 1.** 下载[onnxruntime](https://github.com/microsoft/onnxruntime/releases)，添加环境变量 - -**Step 2.** Inference - -``` +推理代码 +``` bash python .\mmdeploy\demo\python\image_classification.py .\work_dir\onnx\resnet\ .\mmclassification\demo\demo.JPEG ``` -#### TensorRT - -**Step 0.** 安装mmdeploy_python - -``` -pip install .\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\python\mmdeploy_python-0.6.0-cp38-none-win_amd64.whl -``` - -**Step 1.** 按照上述转模型的要求，安装好CUDA Toolkit 11.1，TensorRT 8.2.3.0，cuDNN 8.2.1.0 并设置好环境变量 - -**Step 2.** Inference +#### TensorRT +推理代码 ``` python .\mmdeploy\demo\python\image_classification.py .\work_dir\trt\resnet\ .\mmclassification\demo\demo.JPEG --device-name cuda ``` @@ -429,116 +296,59 @@ pip install .\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\python\m 这里介绍如何使用SDK的C API进行推理 -这里涉及到编译，需要安装Vs2019+，CMake - -example中读取图片用到了OpenCV，所以需要从这里安装[opencv-4.6.0-vc14_vc15.exe](https://github.com/opencv/opencv/releases)，或者自行编译 - -``` -// Current working directories -. -|-- opencv -|-- mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0 -|-- mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1 -|-- mmclassification -|-- mmdeploy -|-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth -`-- work_dir -``` - #### ONNXRuntime -**Step 0.** 编译 - -在mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\\sdk\\example目录下 - -``` -// 部分路径根据所在硬盘的位置进行修改 -mkdir build -cd build -cmake .. -A x64 -T v142 ` - -DOpenCV_DIR=C:\workspace\opencv\build\x64\vc15\lib ` - -DMMDeploy_DIR=C:\workspace\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\lib\cmake\MMDeploy ` - -DONNXRUNTIME_DIR=C:\Deps\onnxruntime\onnxruntime-win-gpu-x64-1.8.1 - -cmake --build . --config Release - -``` - -**Step 1.** 添加环境变量 - -需要添加的环境变量有三个：分别是OpenCV的bin目录，mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\\sdk\\bin，以及onnxruntime的lib目录。在之前已经添加过的变量可以忽略，我这里是： - -``` -C:\Deps\onnxruntime\onnxruntime-win-gpu-x64-1.8.1\lib -C:\workspace\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\bin -C:\workspace\opencv\build\x64\vc15\bin -``` - -这里也可以不添加环境变量，而将这三者的dll拷贝到刚才编译出的exe(build/Release)的同级目录下。 - -**Step 2.** Inference -重启Powershell让环境变量生效。这里建议使用cmd，这样如果exe运行时如果找不到相关的dll的话会有弹窗 -在mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\\sdk\\example\\build\\Release目录下： - -``` -.\image_classification.exe cpu C:\workspace\work_dir\onnx\resnet\ C:\workspace\mmclassification\demo\demo.JPEG -``` - -#### TensorRT - -**Step 0.** 编译 - -在mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\\sdk\\example目录下 - -``` -// 部分路径根据所在硬盘的位置进行修改 -mkdir build -cd build -cmake .. -A x64 -T v142 ` - -DOpenCV_DIR=C:\workspace\opencv\build\x64\vc15\lib ` - -DMMDeploy_DIR=C:\workspace\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\lib\cmake\MMDeploy ` - -DTENSORRT_DIR=C:\Deps\tensorrt\TensorRT-8.2.3.0 ` - -DCUDNN_DIR=C:\Deps\cudnn\8.2.1 - -cmake --build . --config Release -``` - -**可能出现的问题：** - -``` -enable_language(CUDA) 报错 - --- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.19044. --- Found CUDA: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.1 (found version "11.1") -CMake Error at C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCompilerId.cmake:491 (message): - No CUDA toolset found. -Call Stack (most recent call first): - C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCompilerId.cmake:6 (CMAKE_DETERMINE_COMPILER_ID_BUILD) - C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCompilerId.cmake:59 (__determine_compiler_id_test) - C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCUDACompiler.cmake:339 (CMAKE_DETERMINE_COMPILER_ID) - C:/workspace/mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0/sdk/lib/cmake/MMDeploy/MMDeployConfig.cmake:27 (enable_language) - CMakeLists.txt:5 (find_package) -``` - -**原因：** CUDA Toolkit 11.1安装在Visual Studio之前，造成VS的插件没有安装。或者VS的版本过新，使得CUDA Toolkit的安装的时候跳过了VS插件的安装 - -**解决方法：** 可以通过手工拷贝插件的方式来解决这个问题。 -我这里的环境是CUDA Toolkit 11.1，vs2022，操作是将`C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\extras\visual_studio_integration\MSBuildExtensions`中的四个文件拷贝到`C:\Software\Microsoft Visual Studio\2022\Community\Msbuild\Microsoft\VC\v170\BuildCustomizations` 目录下。具体路径根据实际情况进行更改。 - -**Step 1.** 添加环境变量 -这里需要添加以下四个变量，根据各自的情况进行调整 - -``` -C:\Deps\cudnn\8.2.1\bin -C:\Deps\tensorrt\TensorRT-8.2.3.0\lib -C:\workspace\opencv\build\x64\vc15\bin -C:\workspace\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\bin -``` - -**Step 2.** Inference -重启Powershell让环境变量生效。这里建议使用cmd，这样如果exe运行时如果找不到相关的dll的话会有弹窗 -在mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\\sdk\\example\\build\\Release目录下： - -``` -.\image_classification.exe cuda C:\workspace\work_dir\trt\resnet C:\workspace\mmclassification\demo\demo.JPEG -``` +1. 编译Examples +在`mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\example`目录下 + ``` + // 部分路径根据实际位置进行修改 + mkdir build + cd build + cmake .. -A x64 -T v142 ` + -DOpenCV_DIR=C:\workspace\opencv\build\x64\vc15\lib ` + -DMMDeploy_DIR=C:\workspace\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\lib\cmake\MMDeploy ` + -DONNXRUNTIME_DIR=C:\Deps\onnxruntime\onnxruntime-win-gpu-x64-1.8.1 + + cmake --build . --config Release + ``` +2. 添加环境变量或拷贝动态库到exe同级目录 + :point_right: 目的是使exe运行时可以正确找到相关dll + 若选择添加环境变量，则将`mmdeploy`的运行时库路径（`mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\bin`）添加到PATH，可参考onnxruntime的添加过程。 + 若选择拷贝动态库，而将bin目录中的dll拷贝到刚才编译出的exe(build/Release)的同级目录下。 + +3. 推理： + 这里建议使用cmd，这样如果exe运行时如果找不到相关的dll的话会有弹窗 + 在mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\example\build\Release目录下： + ``` + .\image_classification.exe cpu C:\workspace\work_dir\onnx\resnet\ C:\workspace\mmclassification\demo\demo.JPEG + ``` + + +#### TensorRT + +1. 编译 + 在mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\example目录下 + ``` + // 部分路径根据所在硬盘的位置进行修改 + mkdir build + cd build + cmake .. -A x64 -T v142 ` + -DOpenCV_DIR=C:\workspace\opencv\build\x64\vc15\lib ` + -DMMDeploy_DIR=C:\workspace\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8 2.3.0\sdk\lib\cmake\MMDeploy ` + -DTENSORRT_DIR=C:\Deps\tensorrt\TensorRT-8.2.3.0 ` + -DCUDNN_DIR=C:\Deps\cudnn\8.2.1 + cmake --build . --config Release + ``` +2. 添加环境变量或拷贝动态库到exe同级目录 + :point_right: 目的是使exe运行时可以正确找到相关dll + 若选择添加环境变量，则将`mmdeploy`的运行时库路径（`mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\bin`）添加到PATH，可参考onnxruntime的添加过程。 + 若选择拷贝动态库，而将bin目录中的dll拷贝到刚才编译出的exe(build/Release)的同级目录下。 +3. 推理 + 在mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\example\build\Release目录下： + ``` + .\image_classification.exe cuda C:\workspace\work_dir\trt\resnet C:\workspace\mmclassification\demo\demo.JPEG + ``` + +## 可能遇到的问题 + +如遇到问题，可参考[FAQ](../faq.md) diff --git a/docs/zh_cn/faq.md b/docs/zh_cn/faq.md index 8b227cc9fc..47632be6e8 100644 --- a/docs/zh_cn/faq.md +++ b/docs/zh_cn/faq.md @@ -50,6 +50,39 @@ print(torch.__file__) ``` +- 编译时enable_language(CUDA) 报错 + ``` + -- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.19044. + -- Found CUDA: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.1 (found version "11.1") + CMake Error at C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCompilerId.cmake:491 (message): + No CUDA toolset found. + Call Stack (most recent call first): + C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCompilerId.cmake:6 (CMAKE_DETERMINE_COMPILER_ID_BUILD) + C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCompilerId.cmake:59 (__determine_compiler_id_test) + C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCUDACompiler.cmake:339 (CMAKE_DETERMINE_COMPILER_ID) + C:/workspace/mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0/sdk/lib/cmake/MMDeploy/MMDeployConfig.cmake:27 (enable_language) + CMakeLists.txt:5 (find_package) + ``` + **原因：** CUDA Toolkit 11.1安装在Visual Studio之前，造成VS的插件没有安装。或者VS的版本过新，使得CUDA Toolkit的安装的时候跳过了VS插件的安装 + + + **解决方法：** 可以通过手工拷贝插件的方式来解决这个问题。比如将`C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\extras\visual_studio_integration\MSBuildExtensions`中的四个文件拷贝到`C:\Software\Microsoft Visual Studio\2022\Community\Msbuild\Microsoft\VC\v170\BuildCustomizations` 目录下。具体路径根据实际情况进行更改。 + + +### ONNX Runtime +- Windows系统下，转模型可视化时以及SDK推理时遇到 + ``` + onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Failed to load library, error code: 193 + ``` + **原因：** 在较新的windows系统中，系统路径下下有两个`onnxruntime.dll`，且会优先加载，造成冲突。 + ``` + C:\Windows\SysWOW64\onnxruntime.dll + C:\Windows\System32\onnxruntime.dll + ``` + **解决方法：** 以下两个方案任选其一 + 1) 将下载的onnxruntime中的lib目录下的dll拷贝到mmdeploy_onnxruntime_ops.dll的同级目录（推荐使用Everything 进行查找） + 1) 将系统路径下的这两个dll改名，使其加载不到，可能涉及到修改文件权限的操作 + ### Pip - pip installed package but could not `import` them. From 65e4ccd3344007c77191754cfd9baf9da0a26f04 Mon Sep 17 00:00:00 2001 From: Xin Chen Date: Thu, 28 Jul 2022 20:56:09 +0800 Subject: [PATCH 4/8] try fix lint --- .../02-how-to-run/prebuilt_package_windows.md | 170 +++++++++++------- docs/zh_cn/faq.md | 17 +- 2 files changed, 118 insertions(+), 69 deletions(-) diff --git a/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md b/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md index 4b0027d020..6669a1e137 100644 --- a/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md +++ b/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md @@ -19,15 +19,17 @@ - [ONNXRuntime](#onnxruntime-2) - [TensorRT](#tensorrt-3) - [可能遇到的问题](#可能遇到的问题) ----- -目前，`MMDeploy`在`Windows`平台下提供`TensorRT`以及`ONNX Runtime`两种预编译包，可以从[Releases](https://github.com/open-mmlab/mmdeploy/releases)获取。 +______________________________________________________________________ + +目前，`MMDeploy`在`Windows`平台下提供`TensorRT`以及`ONNX Runtime`两种预编译包，可以从[Releases](https://github.com/open-mmlab/mmdeploy/releases)获取。 本篇教程以`mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1.zip`和`mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0.zip`为例，展示预编译包的使用方法。 为了方便使用者快速上手，本教程以分类模型(mmclassification)为例，展示两种预编译包的使用方法。 预编译包的目录结构如下，其中`dist`文件夹为模型转换相关内容，`sdk`文件夹为模型推理相关内容。 + ``` . |-- dist @@ -48,22 +50,31 @@ 首先新建一个工作目录workspace 1. 请按照[get_started](../get_started.md)文档，准备虚拟环境，安装pytorch、torchvision、mmcv-full。若要使用SDK的C接口，需要安装vs2019+, OpenCV。 - :point_right: 这里建议使用`pip`而不是`conda`安装pytorch、torchvision + + :point_right: 这里建议使用`pip`而不是`conda`安装pytorch、torchvision + 2. 克隆mmdeploy仓库 - ``` bash + + ```bash git clone https://github.com/open-mmlab/mmdeploy.git ``` + :point_right: 这里主要为了使用configs文件，所以没有加`--recursive`来下载submodule，也不需要编译`mmdeploy` -3. 安装mmclassification - ``` bash - git clone https://github.com/open-mmlab/mmclassification.git - cd mmclassification - pip install -e . - ``` -4. 准备一个PyTorch的模型文件当作我们的示例 + +3. 安装mmclassification + + ```bash + git clone https://github.com/open-mmlab/mmclassification.git + cd mmclassification + pip install -e . + ``` + +4. 准备一个PyTorch的模型文件当作我们的示例 + 这里选择了[resnet18_8xb32_in1k_20210831-fbbb1da6.pth](https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_8xb32_in1k_20210831-fbbb1da6.pth)，对应的训练config为[resnet18_8xb32_in1k.py](https://github.com/open-mmlab/mmclassification/blob/master/configs/resnet/resnet18_8xb32_in1k.py) 做好以上工作后，当前工作目录的结构应为： + ``` . |-- mmclassification @@ -72,59 +83,68 @@ |-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth ``` - - ### ONNX Runtime 本节介绍`mmdeploy`使用`ONNX Runtime`推理所特有的环境准备工作 5. 安装`mmdeploy`（模型转换）以及`mmdeploy_python`（模型推理Python API）的预编译包 - ``` bash + + ```bash # 先下载 mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1.zip pip install .\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\dist\mmdeploy-0.6.0-py38-none-win_amd64.whl pip install .\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\python\mmdeploy_python-0.6.0-cp38-none-win_amd64.whl ``` + :point_right: 如果之前安装过，需要先卸载后再安装。 + 6. 安装onnxruntime package + ``` pip install onnxruntime==1.8.1 ``` -7. 下载[`onnxruntime`](https://github.com/microsoft/onnxruntime/releases/tag/v1.8.1)，添加环境变量 + +7. 下载[`onnxruntime`](https://github.com/microsoft/onnxruntime/releases/tag/v1.8.1)，添加环境变量 将onnxruntime的lib目录添加到PATH里面，如图所示，具体的路径根据个人情况更改。 - - ![sys-path](https://user-images.githubusercontent.com/16019484/181463801-1d7814a8-b256-46e9-86f2-c08de0bc150b.png) - :exclamation: 重启powershell让环境变量生效，可以通过 echo $env:PATH 来检查是否设置成功。 + ![sys-path](https://user-images.githubusercontent.com/16019484/181463801-1d7814a8-b256-46e9-86f2-c08de0bc150b.png) + :exclamation: 重启powershell让环境变量生效，可以通过 echo $env:PATH 来检查是否设置成功。 ### TensorRT 本节介绍`mmdeploy`使用`TensorRT`推理所特有的环境准备工作 5. 安装`mmdeploy`（模型转换）以及`mmdeploy_python`（模型推理Python API）的预编译包 - ``` bash + + ```bash # 先下载 mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0.zip pip install .\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\dist\mmdeploy-0.6.0-py38-none-win_amd64.whl pip install .\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\python\mmdeploy_python-0.6.0-cp38-none-win_amd64.whl ``` - :point_right: 如果之前安装过，需要先卸载后再安装 -6. 安装CUDA相关内容，并设置环境变量 + + :point_right: 如果之前安装过，需要先卸载后再安装 + +6. 安装CUDA相关内容，并设置环境变量 + - CUDA Toolkit 11.1 - TensorRT 8.2.3.0 (python包 + 环境变量) - cuDNN 8.2.1.0 - - 其中CUDA的环境变量在安装CUDA Toolkit后会自动添加，TensorRT以及cuDNN解压后需要自行添加运行库的路径到PATH，可参考onnxruntime的设置图例 - :exclamation: 重启powershell让环境变量生效，可以通过 echo $env:PATH 来检查是否设置成功 + + 其中CUDA的环境变量在安装CUDA Toolkit后会自动添加，TensorRT以及cuDNN解压后需要自行添加运行库的路径到PATH，可参考onnxruntime的设置图例 + + :exclamation: 重启powershell让环境变量生效，可以通过 echo $env:PATH 来检查是否设置成功 + :exclamation: 建议只添加一个版本的TensorRT的lib到PATH里面。不建议拷贝TensorRT的dll到C盘的cuda目录，在某些情况下，这样可以暴露dll的版本问题 + 7. 安装pycuda `pip install pycuda` ## 模型转换 - ### ONNX Runtime Example 下面介绍根据之前下载的ckpt来展示如果使用`mmdeploy`预编译包来进行模型转换 经过之前的准备工作，当前的工作目录结构应该为： + ``` .. |-- mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1 @@ -135,6 +155,7 @@ ``` python 转换代码 + ```python from mmdeploy.apis import torch2onnx from mmdeploy.backend.sdk.export_info import export2SDK @@ -156,7 +177,8 @@ export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint) ``` 转换后的模型目录结构应该为： -``` bash + +```bash .\work_dir\ `-- onnx `-- resnet @@ -165,6 +187,7 @@ export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint) |-- end2end.onnx `-- pipeline.json ``` + ### TensorRT Example 下面根据之前下载的ckpt来展示如果使用mmdeploy预编译包来进行模型转换 @@ -181,7 +204,8 @@ export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint) ``` python 转换代码 -``` python + +```python from mmdeploy.apis import torch2onnx from mmdeploy.apis.tensorrt import onnx2tensorrt from mmdeploy.backend.sdk.export_info import export2SDK @@ -211,6 +235,7 @@ export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint) ``` 转换后的模型目录结构应该为： + ``` .\work_dir\ `-- trt @@ -225,12 +250,14 @@ export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint) ## 模型推理 以下内容假定已完成了上述模型转换的两个Example，并得到了上述模型转换后的两个文件夹其中之一或者全部： + ``` .\work_dir\onnx\resnet .\work_dir\trt\resnet ``` 当前的工作目录应为： + ``` . |-- mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0 @@ -244,12 +271,13 @@ export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint) ### Backend Inference -:exclamation: 需要强调的一点是，这个接口不是为了做部署的，而是屏蔽了推理后端接口的，用来检验转换的模型是否可以正常推理的。 +:exclamation: 需要强调的一点是，这个接口不是为了做部署的，而是屏蔽了推理后端接口的，用来检验转换的模型是否可以正常推理的。 #### ONNXRuntime Python 代码 -``` python + +```python from mmdeploy.apis import inference_model model_cfg = 'mmclassification/configs/resnet/resnet18_8xb32_in1k.py' @@ -260,10 +288,11 @@ device = 'cpu' result = inference_model(model_cfg, deploy_cfg, backend_files, img, device) ``` -#### TensorRT +#### TensorRT Python 代码 -``` python + +```python from mmdeploy.apis import inference_model model_cfg = 'mmclassification/configs/resnet/resnet18_8xb32_in1k.py' @@ -281,13 +310,15 @@ result = inference_model(model_cfg, deploy_cfg, backend_files, img, device) #### ONNXRuntime 推理代码 -``` bash + +```bash python .\mmdeploy\demo\python\image_classification.py .\work_dir\onnx\resnet\ .\mmclassification\demo\demo.JPEG ``` -#### TensorRT +#### TensorRT 推理代码 + ``` python .\mmdeploy\demo\python\image_classification.py .\work_dir\trt\resnet\ .\mmclassification\demo\demo.JPEG --device-name cuda ``` @@ -298,36 +329,44 @@ python .\mmdeploy\demo\python\image_classification.py .\work_dir\onnx\resnet\ .\ #### ONNXRuntime -1. 编译Examples -在`mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\example`目录下 - ``` - // 部分路径根据实际位置进行修改 - mkdir build - cd build - cmake .. -A x64 -T v142 ` - -DOpenCV_DIR=C:\workspace\opencv\build\x64\vc15\lib ` - -DMMDeploy_DIR=C:\workspace\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\lib\cmake\MMDeploy ` - -DONNXRUNTIME_DIR=C:\Deps\onnxruntime\onnxruntime-win-gpu-x64-1.8.1 - - cmake --build . --config Release - ``` -2. 添加环境变量或拷贝动态库到exe同级目录 - :point_right: 目的是使exe运行时可以正确找到相关dll - 若选择添加环境变量，则将`mmdeploy`的运行时库路径（`mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\bin`）添加到PATH，可参考onnxruntime的添加过程。 +1. 编译Examples + 在`mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\example`目录下 + + ``` + // 部分路径根据实际位置进行修改 + mkdir build + cd build + cmake .. -A x64 -T v142 ` + -DOpenCV_DIR=C:\workspace\opencv\build\x64\vc15\lib ` + -DMMDeploy_DIR=C:\workspace\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\lib\cmake\MMDeploy ` + -DONNXRUNTIME_DIR=C:\Deps\onnxruntime\onnxruntime-win-gpu-x64-1.8.1 + + cmake --build . --config Release + ``` + +2. 添加环境变量或拷贝动态库到exe同级目录 + + :point_right: 目的是使exe运行时可以正确找到相关dll + + 若选择添加环境变量，则将`mmdeploy`的运行时库路径（`mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\bin`）添加到PATH，可参考onnxruntime的添加过程。 + 若选择拷贝动态库，而将bin目录中的dll拷贝到刚才编译出的exe(build/Release)的同级目录下。 -3. 推理： - 这里建议使用cmd，这样如果exe运行时如果找不到相关的dll的话会有弹窗 - 在mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\example\build\Release目录下： +3. 推理： + + 这里建议使用cmd，这样如果exe运行时如果找不到相关的dll的话会有弹窗 + + 在mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\\sdk\\example\\build\\Release目录下： + ``` .\image_classification.exe cpu C:\workspace\work_dir\onnx\resnet\ C:\workspace\mmclassification\demo\demo.JPEG ``` +#### TensorRT -#### TensorRT +1. 编译 + 在mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\\sdk\\example目录下 -1. 编译 - 在mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\example目录下 ``` // 部分路径根据所在硬盘的位置进行修改 mkdir build @@ -336,15 +375,24 @@ python .\mmdeploy\demo\python\image_classification.py .\work_dir\onnx\resnet\ .\ -DOpenCV_DIR=C:\workspace\opencv\build\x64\vc15\lib ` -DMMDeploy_DIR=C:\workspace\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8 2.3.0\sdk\lib\cmake\MMDeploy ` -DTENSORRT_DIR=C:\Deps\tensorrt\TensorRT-8.2.3.0 ` - -DCUDNN_DIR=C:\Deps\cudnn\8.2.1 + -DCUDNN_DIR=C:\Deps\cudnn\8.2.1 cmake --build . --config Release ``` -2. 添加环境变量或拷贝动态库到exe同级目录 - :point_right: 目的是使exe运行时可以正确找到相关dll - 若选择添加环境变量，则将`mmdeploy`的运行时库路径（`mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\bin`）添加到PATH，可参考onnxruntime的添加过程。 + +2. 添加环境变量或拷贝动态库到exe同级目录 + + :point_right: 目的是使exe运行时可以正确找到相关dll + + 若选择添加环境变量，则将`mmdeploy`的运行时库路径（`mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\bin`）添加到PATH，可参考onnxruntime的添加过程。 + 若选择拷贝动态库，而将bin目录中的dll拷贝到刚才编译出的exe(build/Release)的同级目录下。 -3. 推理 - 在mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\example\build\Release目录下： + +3. 推理 + + 这里建议使用cmd，这样如果exe运行时如果找不到相关的dll的话会有弹窗 + + 在mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\\sdk\\example\\build\\Release目录下： + ``` .\image_classification.exe cuda C:\workspace\work_dir\trt\resnet C:\workspace\mmclassification\demo\demo.JPEG ``` diff --git a/docs/zh_cn/faq.md b/docs/zh_cn/faq.md index 47632be6e8..67615b1d14 100644 --- a/docs/zh_cn/faq.md +++ b/docs/zh_cn/faq.md @@ -50,7 +50,8 @@ print(torch.__file__) ``` -- 编译时enable_language(CUDA) 报错 +- 编译时enable_language(CUDA) 报错 + ``` -- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.19044. -- Found CUDA: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.1 (found version "11.1") @@ -62,26 +63,26 @@ C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCUDACompiler.cmake:339 (CMAKE_DETERMINE_COMPILER_ID) C:/workspace/mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0/sdk/lib/cmake/MMDeploy/MMDeployConfig.cmake:27 (enable_language) CMakeLists.txt:5 (find_package) - ``` - **原因：** CUDA Toolkit 11.1安装在Visual Studio之前，造成VS的插件没有安装。或者VS的版本过新，使得CUDA Toolkit的安装的时候跳过了VS插件的安装 + ``` + **原因：** CUDA Toolkit 11.1安装在Visual Studio之前，造成VS的插件没有安装。或者VS的版本过新，使得CUDA Toolkit的安装的时候跳过了VS插件的安装 **解决方法：** 可以通过手工拷贝插件的方式来解决这个问题。比如将`C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\extras\visual_studio_integration\MSBuildExtensions`中的四个文件拷贝到`C:\Software\Microsoft Visual Studio\2022\Community\Msbuild\Microsoft\VC\v170\BuildCustomizations` 目录下。具体路径根据实际情况进行更改。 - ### ONNX Runtime + - Windows系统下，转模型可视化时以及SDK推理时遇到 ``` onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Failed to load library, error code: 193 - ``` + ``` **原因：** 在较新的windows系统中，系统路径下下有两个`onnxruntime.dll`，且会优先加载，造成冲突。 ``` C:\Windows\SysWOW64\onnxruntime.dll C:\Windows\System32\onnxruntime.dll ``` - **解决方法：** 以下两个方案任选其一 - 1) 将下载的onnxruntime中的lib目录下的dll拷贝到mmdeploy_onnxruntime_ops.dll的同级目录（推荐使用Everything 进行查找） - 1) 将系统路径下的这两个dll改名，使其加载不到，可能涉及到修改文件权限的操作 + **解决方法：** 以下两个方案任选其一 + 1. 将下载的onnxruntime中的lib目录下的dll拷贝到mmdeploy_onnxruntime_ops.dll的同级目录（推荐使用Everything 进行查找） + 2. 将系统路径下的这两个dll改名，使其加载不到，可能涉及到修改文件权限的操作 ### Pip From 9ef34d84714d572641eb9361466c372801fcdaeb Mon Sep 17 00:00:00 2001 From: Xin Chen Date: Tue, 2 Aug 2022 15:35:50 +0800 Subject: [PATCH 5/8] add en docs --- .../02-how-to-run/prebuilt_package_windows.md | 404 ++++++++++++++++++ docs/en/faq.md | 34 ++ .../02-how-to-run/prebuilt_package_windows.md | 16 +- 3 files changed, 445 insertions(+), 9 deletions(-) create mode 100644 docs/en/02-how-to-run/prebuilt_package_windows.md diff --git a/docs/en/02-how-to-run/prebuilt_package_windows.md b/docs/en/02-how-to-run/prebuilt_package_windows.md new file mode 100644 index 0000000000..dce554588a --- /dev/null +++ b/docs/en/02-how-to-run/prebuilt_package_windows.md @@ -0,0 +1,404 @@ +# How to use prebuilt package on Windows10 + +- [How to use prebuilt package on Windows10](#how-to-use-prebuilt-package-on-windows10) + - [Prerequisite](#prerequisite) + - [ONNX Runtime](#onnx-runtime) + - [TensorRT](#tensorrt) + - [Model Convert](#model-convert) + - [ONNX Runtime Example](#onnx-runtime-example) + - [TensorRT Example](#tensorrt-example) + - [Model Inference](#model-inference) + - [Backend Inference](#backend-inference) + - [ONNXRuntime](#onnxruntime) + - [TensorRT](#tensorrt-1) + - [Python SDK](#python-sdk) + - [ONNXRuntime](#onnxruntime-1) + - [TensorRT](#tensorrt-2) + - [C SDK](#c-sdk) + - [ONNXRuntime](#onnxruntime-2) + - [TensorRT](#tensorrt-3) + - [Possible problems](#possible-problems) + +______________________________________________________________________ + +Currently, `MMDeploy` provides prebuilt packages with `TensorRT` or `ONNX Runtime` as inference backend on `Windows`. You can download from [Releases](https://github.com/open-mmlab/mmdeploy/releases) page. + +This tutorial takes `mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1.zip` and `mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0.zip` as examples to show how to use the prebuilt packages. + +In order to facilitate users to get started quickly, this tutorial takes the classification model (mmclassification) as an example to show the use of two prebuilt packages. + +The directory structure of the prebuilt package is as follows, where the `dist` folder is the model conversion related content, and the `sdk` folder is the model inference related content. + +``` +. +|-- dist +`-- sdk + |-- bin + |-- example + |-- include + |-- lib + `-- python +``` + +## Prerequisite + +To use the prebuilt package to perform `model conversion` and `model inference`, you need to install some third-party dependent libraries. The following introduces preparations for using `ONNX Runtime` and `TensorRT` as the inference backend. + +In the environment preparation of the two inference backend, some operations are common. The following describes these common operations first, and then introduces their own unique operations. + +First create a new working directory workspace + +Please follow the [get_started](../get_started.md) documentation to + +1. Follow the [get_started](../get_started.md) documentation to create a virtual python environment and install pytorch, torchvision, and mmcv-full. To use the C interface of the SDK, you need to install vs2019+, OpenCV. + + :point_right: It is recommended to use `pip` instead of `conda` to install pytorch, torchvision + +2. Clone the mmdeploy repository + + ```bash + git clone https://github.com/open-mmlab/mmdeploy.git + ``` + + :point_right: The main purpose here is to use the configs, so there is no need to add `--recursive` to download the submodule. It's also no need to compile `mmdeploy`. + +3. Install mmclassification + + ```bash + git clone https://github.com/open-mmlab/mmclassification.git + cd mmclassification + pip install -e . + ``` + +4. Prepare a PyTorch model as our example + + Download the pth [resnet18_8xb32_in1k_20210831-fbbb1da6.pth](https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_8xb32_in1k_20210831-fbbb1da6.pth). The corresponding config is [resnet18_8xb32_in1k.py](https://github.com/open-mmlab/mmclassification/blob/master/configs/resnet/resnet18_8xb32_in1k.py) + +After doing the above work, the structure of the current working directory should be: + +``` +. +|-- mmclassification +|-- mmdeploy +|-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth +``` + +### ONNX Runtime + +This section describes the environment preparations specific to `mmdeploy` for using `ONNX Runtime` as inference backend. + +Install precompiled packages of `mmdeploy` (model transformation) and `mmdeploy_python` (model inference Python API) + +5. Install prebuilt package for `mmdeploy` (Model Convert) and `mmdeploy_python` (Model Inferece Python API). + + ```bash + # download mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1.zip + pip install .\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\dist\mmdeploy-0.6.0-py38-none-win_amd64.whl + pip install .\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\python\mmdeploy_python-0.6.0-cp38-none-win_amd64.whl + ``` + + :point_right: If you have installed it before, please uninstall it first. + +6. Install onnxruntime package + + ``` + pip install onnxruntime==1.8.1 + ``` + +7. Download [`onnxruntime`](https://github.com/microsoft/onnxruntime/releases/tag/v1.8.1), and add environment variable. + + Add the lib directory of onnxruntime to the PATH, as shown in the figure, the specific path should be changed according to individual circumstances. + + ![sys-path](https://user-images.githubusercontent.com/16019484/181463801-1d7814a8-b256-46e9-86f2-c08de0bc150b.png) + :exclamation: Restart powershell to make the environment variables take effect. You can check whether the settings are in effect by echo $env:PATH. + +### TensorRT + +This section describes the environment preparations specific to `mmdeploy` for using `TensorRT` as inference backend. + +5. Install prebuilt package for `mmdeploy` (Model Convert) and `mmdeploy_python` (Model Inferece Python API). + + ```bash + # download mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0.zip + pip install .\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\dist\mmdeploy-0.6.0-py38-none-win_amd64.whl + pip install .\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\python\mmdeploy_python-0.6.0-cp38-none-win_amd64.whl + ``` + + :point_right: If you have installed it before, please uninstall it first. + +6. Install CUDA related content and set environment variables + + - CUDA Toolkit 11.1 + - TensorRT 8.2.3.0 (python package + environment variables) + - cuDNN 8.2.1.0 + + The CUDA environment variable will be automatically added after installing the CUDA Toolkit. For TensorRT and cuDNN, after decompressing the zip package, you need to add the path of the runtime library to the PATH. You can refer to the path setting of onnxruntime. + + :exclamation: Restart powershell to make the environment variables take effect. You can check whether the settings are in effect by echo $env:PATH. + + :exclamation: It is recommended to add only one version of the TensorRT lib to the PATH. It is not recommended to copy the dll of TensorRT to the cuda directory in C drive. In some cases, this can expose the version problem when running the executable program. + +7. Install pycuda by `pip install pycuda` + +## Model Convert + +### ONNX Runtime Example + +The following describes how to use the `mmdeploy` prebuilt package to perform model conversion based on the previously downloaded ckpt. + +After doing the above work, the structure of the current working directory should be： + +``` +.. +|-- mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1 +|-- mmclassification +|-- mmdeploy +`-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth +``` + +The python to convert the model + +```python +from mmdeploy.apis import torch2onnx +from mmdeploy.backend.sdk.export_info import export2SDK + +img = 'mmclassification/demo/demo.JPEG' +work_dir = 'work_dir/onnx/resnet' +save_file = 'end2end.onnx' +deploy_cfg = 'mmdeploy/configs/mmcls/classification_onnxruntime_dynamic.py' +model_cfg = 'mmclassification/configs/resnet/resnet18_8xb32_in1k.py' +model_checkpoint = 'resnet18_8xb32_in1k_20210831-fbbb1da6.pth' +device = 'cpu' + +# 1. convert model to onnx +torch2onnx(img, work_dir, save_file, deploy_cfg, model_cfg, + model_checkpoint, device) + +# 2. extract pipeline info for sdk use (dump-info) +export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint) +``` + +The structure of the converted model directory: + +```bash +.\work_dir\ +`-- onnx + `-- resnet + |-- deploy.json + |-- detail.json + |-- end2end.onnx + `-- pipeline.json +``` + +### TensorRT Example + +The following describes how to use the `mmdeploy` prebuilt package to perform model conversion based on the previously downloaded ckpt. + +After doing the above work, the structure of the current working directory should be： + +``` +.. +|-- mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0 +|-- mmclassification +|-- mmdeploy +`-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth +``` + +The python to convert the model + +```python +from mmdeploy.apis import torch2onnx +from mmdeploy.apis.tensorrt import onnx2tensorrt +from mmdeploy.backend.sdk.export_info import export2SDK +import os + +img = 'mmclassification/demo/demo.JPEG' +work_dir = 'work_dir/trt/resnet' +save_file = 'end2end.onnx' +deploy_cfg = 'mmdeploy/configs/mmcls/classification_tensorrt_static-224x224.py' +model_cfg = 'mmclassification/configs/resnet/resnet18_8xb32_in1k.py' +model_checkpoint = 'resnet18_8xb32_in1k_20210831-fbbb1da6.pth' +device = 'cpu' + +# 1. convert model to IR(onnx) +torch2onnx(img, work_dir, save_file, deploy_cfg, model_cfg, + model_checkpoint, device) + +# 2. convert IR to tensorrt +onnx_model = os.path.join(work_dir, save_file) +save_file = 'end2end.engine' +model_id = 0 +device = 'cuda' +onnx2tensorrt(work_dir, save_file, model_id, deploy_cfg, onnx_model, device) + +# 3. extract pipeline info for sdk use (dump-info) +export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint) +``` + +The structure of the converted model directory: + +``` +.\work_dir\ +`-- trt + `-- resnet + |-- deploy.json + |-- detail.json + |-- end2end.engine + |-- end2end.onnx + `-- pipeline.json +``` + +## Model Inference + +The following content assumes that the two examples of the above model conversion have been completed, and one or both of the two model folders after the above model conversion have been obtained: + +``` +.\work_dir\onnx\resnet +.\work_dir\trt\resnet +``` + +The structure of current working directory： + +``` +. +|-- mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0 +|-- mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1 +|-- mmclassification +|-- mmdeploy +|-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth +`-- work_dir +``` + +### Backend Inference + +:exclamation: It should be emphasized that this interface is not for deployment, but shields the difference of backend interface api. The main purpose of this api is to check whether the converted model can be inferred normally. + +#### ONNXRuntime + +The Python code to inference model + +```python +from mmdeploy.apis import inference_model + +model_cfg = 'mmclassification/configs/resnet/resnet18_8xb32_in1k.py' +deploy_cfg = 'mmdeploy/configs/mmcls/classification_onnxruntime_dynamic.py' +backend_files = ['work_dir/onnx/resnet/end2end.onnx'] +img = 'mmclassification/demo/demo.JPEG' +device = 'cpu' +result = inference_model(model_cfg, deploy_cfg, backend_files, img, device) +``` + +#### TensorRT + +The Python code to inference model + +```python +from mmdeploy.apis import inference_model + +model_cfg = 'mmclassification/configs/resnet/resnet18_8xb32_in1k.py' +deploy_cfg = 'mmdeploy/configs/mmcls/classification_tensorrt_static-224x224.py' +backend_files = ['work_dir/trt/resnet/end2end.engine'] +img = 'mmclassification/demo/demo.JPEG' +device = 'cuda' +result = inference_model(model_cfg, deploy_cfg, backend_files, img, device) +``` + +### Python SDK + +The following describes how to use the SDK's Python API for inference + +#### ONNXRuntime + +The inference code + +```bash +python .\mmdeploy\demo\python\image_classification.py .\work_dir\onnx\resnet\ .\mmclassification\demo\demo.JPEG +``` + +#### TensorRT + +The inference code + +``` + python .\mmdeploy\demo\python\image_classification.py .\work_dir\trt\resnet\ .\mmclassification\demo\demo.JPEG --device-name cuda +``` + +### C SDK + +The following describes how to use the SDK's C API for inference + +#### ONNXRuntime + +1. Build examples + + Under `mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\example` directory + + ``` + // Path should be modified according to the actual location + mkdir build + cd build + cmake .. -A x64 -T v142 ` + -DOpenCV_DIR=C:\Deps\opencv\build\x64\vc15\lib ` + -DMMDeploy_DIR=C:\workspace\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\lib\cmake\MMDeploy ` + -DONNXRUNTIME_DIR=C:\Deps\onnxruntime\onnxruntime-win-gpu-x64-1.8.1 + + cmake --build . --config Release + ``` + +2. Add environment variables or copy the runtime dll to the level directory of exe + + :point_right: The purpose is to make the exe find the relevant dll + + If choose to add environment variables, add the runtime library path of `mmdeploy` (`mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\bin`) to the PATH. You can refer to the process of adding onnxruntime lib path. + + If choose to copy the dynamic library, copy the dll in the bin directory to the same level directory of the just compiled exe (build/Release). + +3. Inference： + + It is recommended to use cmd here. It would pop-up a window showing the missing dll if you didn't set the path right. + + Under `mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\\sdk\\example\\build\\Release` directory： + + ``` + .\image_classification.exe cpu C:\workspace\work_dir\onnx\resnet\ C:\workspace\mmclassification\demo\demo.JPEG + ``` + +#### TensorRT + +1. Build examples + + Under `mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\\sdk\\example` directory + + ``` + // Path should be modified according to the actual location + mkdir build + cd build + cmake .. -A x64 -T v142 ` + -DOpenCV_DIR=C:\Deps\opencv\build\x64\vc15\lib ` + -DMMDeploy_DIR=C:\workspace\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8 2.3.0\sdk\lib\cmake\MMDeploy ` + -DTENSORRT_DIR=C:\Deps\tensorrt\TensorRT-8.2.3.0 ` + -DCUDNN_DIR=C:\Deps\cudnn\8.2.1 + cmake --build . --config Release + ``` + +2. Add environment variables or copy the runtime dll to the level directory of exe + + :point_right: The purpose is to make the exe find the relevant dll + + If choose to add environment variables, add the runtime library path of `mmdeploy` (`mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\bin`) to the PATH. You can refer to the process of adding onnxruntime lib path. + + If choose to copy the dynamic library, copy the dll in the bin directory to the same level directory of the just compiled exe (build/Release). + +3. Inference + + It is recommended to use cmd here. It would pop-up a window showing the missing dll if you didn't set the path right. + + Under `mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\\sdk\\example\\build\\Release` directory + + ``` + .\image_classification.exe cuda C:\workspace\work_dir\trt\resnet C:\workspace\mmclassification\demo\demo.JPEG + ``` + +## Possible problems + +If you encounter problems, please refer to [FAQ](../faq.md) diff --git a/docs/en/faq.md b/docs/en/faq.md index dec0174d3c..6230ff8cf4 100644 --- a/docs/en/faq.md +++ b/docs/en/faq.md @@ -50,6 +50,40 @@ print(torch.__file__) ``` +- enable_language(CUDA) error + + ``` + -- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.19044. + -- Found CUDA: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.1 (found version "11.1") + CMake Error at C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCompilerId.cmake:491 (message): + No CUDA toolset found. + Call Stack (most recent call first): + C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCompilerId.cmake:6 (CMAKE_DETERMINE_COMPILER_ID_BUILD) + C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCompilerId.cmake:59 (__determine_compiler_id_test) + C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCUDACompiler.cmake:339 (CMAKE_DETERMINE_COMPILER_ID) + C:/workspace/mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0/sdk/lib/cmake/MMDeploy/MMDeployConfig.cmake:27 (enable_language) + CMakeLists.txt:5 (find_package) + ``` + + **Cause：** CUDA Toolkit 11.1 was installed before Visual Studio, so the VS plugin was not installed. Or the version of VS is too new, so that the installation of the VS plugin is skipped during the installation of the CUDA Toolkit + + **Solution：** This problem can be solved by manually copying the plugin。Like copy the four files in `C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\extras\visual_studio_integration\MSBuildExtensions` to `C:\Software\Microsoft Visual Studio\2022\Community\Msbuild\Microsoft\VC\v170\BuildCustomizations` The specific path should be changed according to the actual situation. + +### ONNX Runtime + +- Under Windows system, when visualize model during model conversion or inference a model + ``` + onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Failed to load library, error code: 193 + ``` + **Cause：** In newer Windows systems, there are two `onnxruntime.dll` under the system path, and they will be loaded first, causing conflicts. + ``` + C:\Windows\SysWOW64\onnxruntime.dll + C:\Windows\System32\onnxruntime.dll + ``` + **Solution：** Choose one of the following two options + 1. Copy the dll in the lib directory of the downloaded onnxruntime to the same level directory as mmdeploy_onnxruntime_ops.dll (It is recommended to use Everything to search the ops dll) + 2. Rename the two dlls in the system path so that they cannot be loaded. + ### Pip - pip installed package but could not `import` them. diff --git a/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md b/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md index 6669a1e137..3bef241310 100644 --- a/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md +++ b/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md @@ -1,7 +1,6 @@ # Win10 下预编译包的使用 - [Win10 下预编译包的使用](#win10-下预编译包的使用) - - [- 可能遇到的问题](#--可能遇到的问题) - [准备工作](#准备工作) - [ONNX Runtime](#onnx-runtime) - [TensorRT](#tensorrt) @@ -79,7 +78,6 @@ ______________________________________________________________________ . |-- mmclassification |-- mmdeploy -|-- opencv |-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth ``` @@ -104,6 +102,7 @@ ______________________________________________________________________ ``` 7. 下载[`onnxruntime`](https://github.com/microsoft/onnxruntime/releases/tag/v1.8.1)，添加环境变量 + 将onnxruntime的lib目录添加到PATH里面，如图所示，具体的路径根据个人情况更改。 ![sys-path](https://user-images.githubusercontent.com/16019484/181463801-1d7814a8-b256-46e9-86f2-c08de0bc150b.png) @@ -150,7 +149,6 @@ ______________________________________________________________________ |-- mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1 |-- mmclassification |-- mmdeploy -|-- opencv `-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth ``` @@ -199,7 +197,6 @@ export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint) |-- mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0 |-- mmclassification |-- mmdeploy -|-- opencv `-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth ``` @@ -264,7 +261,6 @@ export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint) |-- mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1 |-- mmclassification |-- mmdeploy -|-- opencv |-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth `-- work_dir ``` @@ -329,7 +325,8 @@ python .\mmdeploy\demo\python\image_classification.py .\work_dir\onnx\resnet\ .\ #### ONNXRuntime -1. 编译Examples +1. 编译 examples + 在`mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\example`目录下 ``` @@ -337,7 +334,7 @@ python .\mmdeploy\demo\python\image_classification.py .\work_dir\onnx\resnet\ .\ mkdir build cd build cmake .. -A x64 -T v142 ` - -DOpenCV_DIR=C:\workspace\opencv\build\x64\vc15\lib ` + -DOpenCV_DIR=C:\Deps\opencv\build\x64\vc15\lib ` -DMMDeploy_DIR=C:\workspace\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\lib\cmake\MMDeploy ` -DONNXRUNTIME_DIR=C:\Deps\onnxruntime\onnxruntime-win-gpu-x64-1.8.1 @@ -364,7 +361,8 @@ python .\mmdeploy\demo\python\image_classification.py .\work_dir\onnx\resnet\ .\ #### TensorRT -1. 编译 +1. 编译 examples + 在mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\\sdk\\example目录下 ``` @@ -372,7 +370,7 @@ python .\mmdeploy\demo\python\image_classification.py .\work_dir\onnx\resnet\ .\ mkdir build cd build cmake .. -A x64 -T v142 ` - -DOpenCV_DIR=C:\workspace\opencv\build\x64\vc15\lib ` + -DOpenCV_DIR=C:\Deps\opencv\build\x64\vc15\lib ` -DMMDeploy_DIR=C:\workspace\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8 2.3.0\sdk\lib\cmake\MMDeploy ` -DTENSORRT_DIR=C:\Deps\tensorrt\TensorRT-8.2.3.0 ` -DCUDNN_DIR=C:\Deps\cudnn\8.2.1 From 1eab8140a4aaacfcdd2efe3763aa60ba43ad6755 Mon Sep 17 00:00:00 2001 From: Xin Chen Date: Tue, 2 Aug 2022 18:26:59 +0800 Subject: [PATCH 6/8] update --- .../02-how-to-run/prebuilt_package_windows.md | 80 ++++++++----------- 1 file changed, 34 insertions(+), 46 deletions(-) diff --git a/docs/en/02-how-to-run/prebuilt_package_windows.md b/docs/en/02-how-to-run/prebuilt_package_windows.md index dce554588a..551a9077e3 100644 --- a/docs/en/02-how-to-run/prebuilt_package_windows.md +++ b/docs/en/02-how-to-run/prebuilt_package_windows.md @@ -17,17 +17,13 @@ - [C SDK](#c-sdk) - [ONNXRuntime](#onnxruntime-2) - [TensorRT](#tensorrt-3) - - [Possible problems](#possible-problems) + - [Troubleshooting](#troubleshooting) ______________________________________________________________________ -Currently, `MMDeploy` provides prebuilt packages with `TensorRT` or `ONNX Runtime` as inference backend on `Windows`. You can download from [Releases](https://github.com/open-mmlab/mmdeploy/releases) page. - This tutorial takes `mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1.zip` and `mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0.zip` as examples to show how to use the prebuilt packages. -In order to facilitate users to get started quickly, this tutorial takes the classification model (mmclassification) as an example to show the use of two prebuilt packages. - -The directory structure of the prebuilt package is as follows, where the `dist` folder is the model conversion related content, and the `sdk` folder is the model inference related content. +The directory structure of the prebuilt package is as follows, where the `dist` folder is about model converter, and the `sdk` folder is related to model inference. ``` . @@ -42,17 +38,11 @@ The directory structure of the prebuilt package is as follows, where the `dist` ## Prerequisite -To use the prebuilt package to perform `model conversion` and `model inference`, you need to install some third-party dependent libraries. The following introduces preparations for using `ONNX Runtime` and `TensorRT` as the inference backend. - -In the environment preparation of the two inference backend, some operations are common. The following describes these common operations first, and then introduces their own unique operations. +In order to use the prebuilt package, you need to install some third-party dependent libraries. -First create a new working directory workspace +1. Follow the [get_started](../get_started.md) documentation to create a virtual python environment and install pytorch, torchvision and mmcv-full. To use the C interface of the SDK, you need to install [vs2019+](https://visualstudio.microsoft.com/), [OpenCV](https://github.com/opencv/opencv/releases). -Please follow the [get_started](../get_started.md) documentation to - -1. Follow the [get_started](../get_started.md) documentation to create a virtual python environment and install pytorch, torchvision, and mmcv-full. To use the C interface of the SDK, you need to install vs2019+, OpenCV. - - :point_right: It is recommended to use `pip` instead of `conda` to install pytorch, torchvision + :point_right: It is recommended to use `pip` instead of `conda` to install pytorch and torchvision 2. Clone the mmdeploy repository @@ -60,7 +50,7 @@ Please follow the [get_started](../get_started.md) documentation to git clone https://github.com/open-mmlab/mmdeploy.git ``` - :point_right: The main purpose here is to use the configs, so there is no need to add `--recursive` to download the submodule. It's also no need to compile `mmdeploy`. + :point_right: The main purpose here is to use the configs, so there is no need to compile `mmdeploy`. 3. Install mmclassification @@ -72,9 +62,9 @@ Please follow the [get_started](../get_started.md) documentation to 4. Prepare a PyTorch model as our example - Download the pth [resnet18_8xb32_in1k_20210831-fbbb1da6.pth](https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_8xb32_in1k_20210831-fbbb1da6.pth). The corresponding config is [resnet18_8xb32_in1k.py](https://github.com/open-mmlab/mmclassification/blob/master/configs/resnet/resnet18_8xb32_in1k.py) + Download the pth [resnet18_8xb32_in1k_20210831-fbbb1da6.pth](https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_8xb32_in1k_20210831-fbbb1da6.pth). The corresponding config of the model is [resnet18_8xb32_in1k.py](https://github.com/open-mmlab/mmclassification/blob/master/configs/resnet/resnet18_8xb32_in1k.py) -After doing the above work, the structure of the current working directory should be: +After the above work is done, the structure of the current working directory should be: ``` . @@ -85,11 +75,9 @@ After doing the above work, the structure of the current working directory shoul ### ONNX Runtime -This section describes the environment preparations specific to `mmdeploy` for using `ONNX Runtime` as inference backend. - -Install precompiled packages of `mmdeploy` (model transformation) and `mmdeploy_python` (model inference Python API) +In order to use `ONNX Runtime` backend, you should also do the following steps. -5. Install prebuilt package for `mmdeploy` (Model Convert) and `mmdeploy_python` (Model Inferece Python API). +5. Install `mmdeploy` (Model Converter) and `mmdeploy_python` (SDK Python API). ```bash # download mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1.zip @@ -107,16 +95,16 @@ Install precompiled packages of `mmdeploy` (model transformation) and `mmdeploy_ 7. Download [`onnxruntime`](https://github.com/microsoft/onnxruntime/releases/tag/v1.8.1), and add environment variable. - Add the lib directory of onnxruntime to the PATH, as shown in the figure, the specific path should be changed according to individual circumstances. + As shown in the figure, add the lib directory of onnxruntime to the PATH. ![sys-path](https://user-images.githubusercontent.com/16019484/181463801-1d7814a8-b256-46e9-86f2-c08de0bc150b.png) - :exclamation: Restart powershell to make the environment variables take effect. You can check whether the settings are in effect by echo $env:PATH. + :exclamation: Restart powershell to make the environment variables setting take effect. You can check whether the settings are in effect by `echo $env:PATH`. ### TensorRT -This section describes the environment preparations specific to `mmdeploy` for using `TensorRT` as inference backend. +In order to use `TensorRT` backend, you should also do the following steps. -5. Install prebuilt package for `mmdeploy` (Model Convert) and `mmdeploy_python` (Model Inferece Python API). +5. Install `mmdeploy` (Model Converter) and `mmdeploy_python` (SDK Python API). ```bash # download mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0.zip @@ -126,17 +114,17 @@ This section describes the environment preparations specific to `mmdeploy` for u :point_right: If you have installed it before, please uninstall it first. -6. Install CUDA related content and set environment variables +6. Install TensorRT related package and set environment variables - CUDA Toolkit 11.1 - - TensorRT 8.2.3.0 (python package + environment variables) + - TensorRT 8.2.3.0 - cuDNN 8.2.1.0 - The CUDA environment variable will be automatically added after installing the CUDA Toolkit. For TensorRT and cuDNN, after decompressing the zip package, you need to add the path of the runtime library to the PATH. You can refer to the path setting of onnxruntime. + Add the runtime libraries of TensorRT and cuDNN to the PATH. You can refer to the path setting of onnxruntime. Don't forget to install python package of TensorRT. - :exclamation: Restart powershell to make the environment variables take effect. You can check whether the settings are in effect by echo $env:PATH. + :exclamation: Restart powershell to make the environment variables setting take effect. You can check whether the settings are in effect by echo `$env:PATH`. - :exclamation: It is recommended to add only one version of the TensorRT lib to the PATH. It is not recommended to copy the dll of TensorRT to the cuda directory in C drive. In some cases, this can expose the version problem when running the executable program. + :exclamation: It is recommended to add only one version of the TensorRT/cuDNN runtime libraries to the PATH. It is better not to copy the runtime libraries of TensorRT/cuDNN to the cuda directory in `C:\`. 7. Install pycuda by `pip install pycuda` @@ -144,9 +132,9 @@ This section describes the environment preparations specific to `mmdeploy` for u ### ONNX Runtime Example -The following describes how to use the `mmdeploy` prebuilt package to perform model conversion based on the previously downloaded ckpt. +The following describes how to use the prebuilt package to do model conversion based on the previous downloaded pth. -After doing the above work, the structure of the current working directory should be： +After the above work is done, the structure of the current working directory should be： ``` .. @@ -192,9 +180,9 @@ The structure of the converted model directory: ### TensorRT Example -The following describes how to use the `mmdeploy` prebuilt package to perform model conversion based on the previously downloaded ckpt. +The following describes how to use the prebuilt package to do model conversion based on the previous downloaded ckpt. -After doing the above work, the structure of the current working directory should be： +After the above work is done, the structure of the current working directory should be： ``` .. @@ -250,7 +238,7 @@ The structure of the converted model directory: ## Model Inference -The following content assumes that the two examples of the above model conversion have been completed, and one or both of the two model folders after the above model conversion have been obtained: +You can obtain two model folders after model conversion. ``` .\work_dir\onnx\resnet @@ -271,7 +259,7 @@ The structure of current working directory： ### Backend Inference -:exclamation: It should be emphasized that this interface is not for deployment, but shields the difference of backend interface api. The main purpose of this api is to check whether the converted model can be inferred normally. +:exclamation: It should be emphasized that this interface is not for deployment, but shields the difference of backend inference api. The main purpose of this api is to check whether the converted model can be inferred normally. #### ONNXRuntime @@ -345,17 +333,17 @@ The following describes how to use the SDK's C API for inference cmake --build . --config Release ``` -2. Add environment variables or copy the runtime dll to the level directory of exe +2. Add environment variables or copy the runtime libraries to the same level directory of exe :point_right: The purpose is to make the exe find the relevant dll - If choose to add environment variables, add the runtime library path of `mmdeploy` (`mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\bin`) to the PATH. You can refer to the process of adding onnxruntime lib path. + If choose to add environment variables, add the runtime libraries path of `mmdeploy` (`mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\bin`) to the PATH. - If choose to copy the dynamic library, copy the dll in the bin directory to the same level directory of the just compiled exe (build/Release). + If choose to copy the dynamic libraries, copy the dll in the bin directory to the same level directory of the just compiled exe (build/Release). 3. Inference： - It is recommended to use cmd here. It would pop-up a window showing the missing dll if you didn't set the path right. + It is recommended to use `CMD` here. Under `mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\\sdk\\example\\build\\Release` directory： @@ -381,17 +369,17 @@ The following describes how to use the SDK's C API for inference cmake --build . --config Release ``` -2. Add environment variables or copy the runtime dll to the level directory of exe +2. Add environment variables or copy the runtime libraries to the same level directory of exe :point_right: The purpose is to make the exe find the relevant dll - If choose to add environment variables, add the runtime library path of `mmdeploy` (`mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\bin`) to the PATH. You can refer to the process of adding onnxruntime lib path. + If choose to add environment variables, add the runtime libraries path of `mmdeploy` (`mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\bin`) to the PATH. - If choose to copy the dynamic library, copy the dll in the bin directory to the same level directory of the just compiled exe (build/Release). + If choose to copy the dynamic libraries, copy the dll in the bin directory to the same level directory of the just compiled exe (build/Release). 3. Inference - It is recommended to use cmd here. It would pop-up a window showing the missing dll if you didn't set the path right. + It is recommended to use `CMD` here. Under `mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\\sdk\\example\\build\\Release` directory @@ -399,6 +387,6 @@ The following describes how to use the SDK's C API for inference .\image_classification.exe cuda C:\workspace\work_dir\trt\resnet C:\workspace\mmclassification\demo\demo.JPEG ``` -## Possible problems +## Troubleshooting If you encounter problems, please refer to [FAQ](../faq.md) From a48f493878498082961fba7bd81d4f94f922e7d9 Mon Sep 17 00:00:00 2001 From: Xin Chen Date: Wed, 3 Aug 2022 13:46:36 +0800 Subject: [PATCH 7/8] update --- .../02-how-to-run/prebuilt_package_windows.md | 30 +++++++------------ 1 file changed, 11 insertions(+), 19 deletions(-) diff --git a/docs/en/02-how-to-run/prebuilt_package_windows.md b/docs/en/02-how-to-run/prebuilt_package_windows.md index 551a9077e3..f14f17889b 100644 --- a/docs/en/02-how-to-run/prebuilt_package_windows.md +++ b/docs/en/02-how-to-run/prebuilt_package_windows.md @@ -95,7 +95,7 @@ In order to use `ONNX Runtime` backend, you should also do the following steps. 7. Download [`onnxruntime`](https://github.com/microsoft/onnxruntime/releases/tag/v1.8.1), and add environment variable. - As shown in the figure, add the lib directory of onnxruntime to the PATH. + As shown in the figure, add the lib directory of onnxruntime to the `PATH`. ![sys-path](https://user-images.githubusercontent.com/16019484/181463801-1d7814a8-b256-46e9-86f2-c08de0bc150b.png) :exclamation: Restart powershell to make the environment variables setting take effect. You can check whether the settings are in effect by `echo $env:PATH`. @@ -120,11 +120,11 @@ In order to use `TensorRT` backend, you should also do the following steps. - TensorRT 8.2.3.0 - cuDNN 8.2.1.0 - Add the runtime libraries of TensorRT and cuDNN to the PATH. You can refer to the path setting of onnxruntime. Don't forget to install python package of TensorRT. + Add the runtime libraries of TensorRT and cuDNN to the `PATH`. You can refer to the path setting of onnxruntime. Don't forget to install python package of TensorRT. :exclamation: Restart powershell to make the environment variables setting take effect. You can check whether the settings are in effect by echo `$env:PATH`. - :exclamation: It is recommended to add only one version of the TensorRT/cuDNN runtime libraries to the PATH. It is better not to copy the runtime libraries of TensorRT/cuDNN to the cuda directory in `C:\`. + :exclamation: It is recommended to add only one version of the TensorRT/cuDNN runtime libraries to the `PATH`. It is better not to copy the runtime libraries of TensorRT/cuDNN to the cuda directory in `C:\`. 7. Install pycuda by `pip install pycuda` @@ -134,7 +134,7 @@ In order to use `TensorRT` backend, you should also do the following steps. The following describes how to use the prebuilt package to do model conversion based on the previous downloaded pth. -After the above work is done, the structure of the current working directory should be： +After preparation work, the structure of the current working directory should be： ``` .. @@ -144,7 +144,7 @@ After the above work is done, the structure of the current working directory sho `-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth ``` -The python to convert the model +Model conversion can be performed like below: ```python from mmdeploy.apis import torch2onnx @@ -180,9 +180,9 @@ The structure of the converted model directory: ### TensorRT Example -The following describes how to use the prebuilt package to do model conversion based on the previous downloaded ckpt. +The following describes how to use the prebuilt package to do model conversion based on the previous downloaded pth. -After the above work is done, the structure of the current working directory should be： +After installation of mmdeploy-tensorrt prebuilt package, the structure of the current working directory should be： ``` .. @@ -192,7 +192,7 @@ After the above work is done, the structure of the current working directory sho `-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth ``` -The python to convert the model +Model conversion can be performed like below: ```python from mmdeploy.apis import torch2onnx @@ -259,12 +259,10 @@ The structure of current working directory： ### Backend Inference -:exclamation: It should be emphasized that this interface is not for deployment, but shields the difference of backend inference api. The main purpose of this api is to check whether the converted model can be inferred normally. +:exclamation: It should be emphasized that `inference_model` is not for deployment, but shields the difference of backend inference api(`TensorRT`, `ONNX Runtime` etc.). The main purpose of this api is to check whether the converted model can be inferred normally. #### ONNXRuntime -The Python code to inference model - ```python from mmdeploy.apis import inference_model @@ -278,8 +276,6 @@ result = inference_model(model_cfg, deploy_cfg, backend_files, img, device) #### TensorRT -The Python code to inference model - ```python from mmdeploy.apis import inference_model @@ -297,16 +293,12 @@ The following describes how to use the SDK's Python API for inference #### ONNXRuntime -The inference code - ```bash python .\mmdeploy\demo\python\image_classification.py .\work_dir\onnx\resnet\ .\mmclassification\demo\demo.JPEG ``` #### TensorRT -The inference code - ``` python .\mmdeploy\demo\python\image_classification.py .\work_dir\trt\resnet\ .\mmclassification\demo\demo.JPEG --device-name cuda ``` @@ -337,7 +329,7 @@ The following describes how to use the SDK's C API for inference :point_right: The purpose is to make the exe find the relevant dll - If choose to add environment variables, add the runtime libraries path of `mmdeploy` (`mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\bin`) to the PATH. + If choose to add environment variables, add the runtime libraries path of `mmdeploy` (`mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\bin`) to the `PATH`. If choose to copy the dynamic libraries, copy the dll in the bin directory to the same level directory of the just compiled exe (build/Release). @@ -373,7 +365,7 @@ The following describes how to use the SDK's C API for inference :point_right: The purpose is to make the exe find the relevant dll - If choose to add environment variables, add the runtime libraries path of `mmdeploy` (`mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\bin`) to the PATH. + If choose to add environment variables, add the runtime libraries path of `mmdeploy` (`mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\bin`) to the `PATH`. If choose to copy the dynamic libraries, copy the dll in the bin directory to the same level directory of the just compiled exe (build/Release). From 7e635c393e419b1859d27dec2ef51f8ddec0f2b4 Mon Sep 17 00:00:00 2001 From: Xin Chen Date: Wed, 3 Aug 2022 14:23:11 +0800 Subject: [PATCH 8/8] udpate faq --- docs/en/faq.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/en/faq.md b/docs/en/faq.md index 6230ff8cf4..1eea9b08e3 100644 --- a/docs/en/faq.md +++ b/docs/en/faq.md @@ -67,21 +67,21 @@ **Cause：** CUDA Toolkit 11.1 was installed before Visual Studio, so the VS plugin was not installed. Or the version of VS is too new, so that the installation of the VS plugin is skipped during the installation of the CUDA Toolkit - **Solution：** This problem can be solved by manually copying the plugin。Like copy the four files in `C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\extras\visual_studio_integration\MSBuildExtensions` to `C:\Software\Microsoft Visual Studio\2022\Community\Msbuild\Microsoft\VC\v170\BuildCustomizations` The specific path should be changed according to the actual situation. + **Solution：** This problem can be solved by manually copying the four files in `C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\extras\visual_studio_integration\MSBuildExtensions` to `C:\Software\Microsoft Visual Studio\2022\Community\Msbuild\Microsoft\VC\v170\BuildCustomizations` The specific path should be changed according to the actual situation. ### ONNX Runtime -- Under Windows system, when visualize model during model conversion or inference a model +- Under Windows system, when visualizing model inference result failed with the following error: ``` onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Failed to load library, error code: 193 ``` - **Cause：** In newer Windows systems, there are two `onnxruntime.dll` under the system path, and they will be loaded first, causing conflicts. + **Cause：** In latest Windows systems, there are two `onnxruntime.dll` under the system path, and they will be loaded first, causing conflicts. ``` C:\Windows\SysWOW64\onnxruntime.dll C:\Windows\System32\onnxruntime.dll ``` **Solution：** Choose one of the following two options - 1. Copy the dll in the lib directory of the downloaded onnxruntime to the same level directory as mmdeploy_onnxruntime_ops.dll (It is recommended to use Everything to search the ops dll) + 1. Copy the dll in the lib directory of the downloaded onnxruntime to the directory where mmdeploy_onnxruntime_ops.dll locates (It is recommended to use Everything to search the ops dll) 2. Rename the two dlls in the system path so that they cannot be loaded. ### Pip