From 091bfc3011ef41b3b4aab1c68c6a713b71396c41 Mon Sep 17 00:00:00 2001 From: Chen Xin Date: Wed, 3 Aug 2022 15:25:36 +0800 Subject: [PATCH] add prebuild package usage docs on windows (#816) * add prebuild package usage docs on windows * fix lint * update * try fix lint * add en docs * update * update * udpate faq --- .../02-how-to-run/prebuilt_package_windows.md | 384 +++++++++++++++++ docs/en/faq.md | 34 ++ .../02-how-to-run/prebuilt_package_windows.md | 400 ++++++++++++++++++ docs/zh_cn/faq.md | 34 ++ 4 files changed, 852 insertions(+) create mode 100644 docs/en/02-how-to-run/prebuilt_package_windows.md create mode 100644 docs/zh_cn/02-how-to-run/prebuilt_package_windows.md diff --git a/docs/en/02-how-to-run/prebuilt_package_windows.md b/docs/en/02-how-to-run/prebuilt_package_windows.md new file mode 100644 index 0000000000..f14f17889b --- /dev/null +++ b/docs/en/02-how-to-run/prebuilt_package_windows.md @@ -0,0 +1,384 @@ +# How to use prebuilt package on Windows10 + +- [How to use prebuilt package on Windows10](#how-to-use-prebuilt-package-on-windows10) + - [Prerequisite](#prerequisite) + - [ONNX Runtime](#onnx-runtime) + - [TensorRT](#tensorrt) + - [Model Convert](#model-convert) + - [ONNX Runtime Example](#onnx-runtime-example) + - [TensorRT Example](#tensorrt-example) + - [Model Inference](#model-inference) + - [Backend Inference](#backend-inference) + - [ONNXRuntime](#onnxruntime) + - [TensorRT](#tensorrt-1) + - [Python SDK](#python-sdk) + - [ONNXRuntime](#onnxruntime-1) + - [TensorRT](#tensorrt-2) + - [C SDK](#c-sdk) + - [ONNXRuntime](#onnxruntime-2) + - [TensorRT](#tensorrt-3) + - [Troubleshooting](#troubleshooting) + +______________________________________________________________________ + +This tutorial takes `mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1.zip` and `mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0.zip` as examples to show how to use the prebuilt packages. + +The directory structure of the prebuilt package is as follows, where the `dist` folder is about model converter, and the `sdk` folder is related to model inference. + +``` +. +|-- dist +`-- sdk + |-- bin + |-- example + |-- include + |-- lib + `-- python +``` + +## Prerequisite + +In order to use the prebuilt package, you need to install some third-party dependent libraries. + +1. Follow the [get_started](../get_started.md) documentation to create a virtual python environment and install pytorch, torchvision and mmcv-full. To use the C interface of the SDK, you need to install [vs2019+](https://visualstudio.microsoft.com/), [OpenCV](https://github.com/opencv/opencv/releases). + + :point_right: It is recommended to use `pip` instead of `conda` to install pytorch and torchvision + +2. Clone the mmdeploy repository + + ```bash + git clone https://github.com/open-mmlab/mmdeploy.git + ``` + + :point_right: The main purpose here is to use the configs, so there is no need to compile `mmdeploy`. + +3. Install mmclassification + + ```bash + git clone https://github.com/open-mmlab/mmclassification.git + cd mmclassification + pip install -e . + ``` + +4. Prepare a PyTorch model as our example + + Download the pth [resnet18_8xb32_in1k_20210831-fbbb1da6.pth](https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_8xb32_in1k_20210831-fbbb1da6.pth). The corresponding config of the model is [resnet18_8xb32_in1k.py](https://github.com/open-mmlab/mmclassification/blob/master/configs/resnet/resnet18_8xb32_in1k.py) + +After the above work is done, the structure of the current working directory should be: + +``` +. +|-- mmclassification +|-- mmdeploy +|-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth +``` + +### ONNX Runtime + +In order to use `ONNX Runtime` backend, you should also do the following steps. + +5. Install `mmdeploy` (Model Converter) and `mmdeploy_python` (SDK Python API). + + ```bash + # download mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1.zip + pip install .\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\dist\mmdeploy-0.6.0-py38-none-win_amd64.whl + pip install .\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\python\mmdeploy_python-0.6.0-cp38-none-win_amd64.whl + ``` + + :point_right: If you have installed it before, please uninstall it first. + +6. Install onnxruntime package + + ``` + pip install onnxruntime==1.8.1 + ``` + +7. Download [`onnxruntime`](https://github.com/microsoft/onnxruntime/releases/tag/v1.8.1), and add environment variable. + + As shown in the figure, add the lib directory of onnxruntime to the `PATH`. + + ![sys-path](https://user-images.githubusercontent.com/16019484/181463801-1d7814a8-b256-46e9-86f2-c08de0bc150b.png) + :exclamation: Restart powershell to make the environment variables setting take effect. You can check whether the settings are in effect by `echo $env:PATH`. + +### TensorRT + +In order to use `TensorRT` backend, you should also do the following steps. + +5. Install `mmdeploy` (Model Converter) and `mmdeploy_python` (SDK Python API). + + ```bash + # download mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0.zip + pip install .\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\dist\mmdeploy-0.6.0-py38-none-win_amd64.whl + pip install .\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\python\mmdeploy_python-0.6.0-cp38-none-win_amd64.whl + ``` + + :point_right: If you have installed it before, please uninstall it first. + +6. Install TensorRT related package and set environment variables + + - CUDA Toolkit 11.1 + - TensorRT 8.2.3.0 + - cuDNN 8.2.1.0 + + Add the runtime libraries of TensorRT and cuDNN to the `PATH`. You can refer to the path setting of onnxruntime. Don't forget to install python package of TensorRT. + + :exclamation: Restart powershell to make the environment variables setting take effect. You can check whether the settings are in effect by echo `$env:PATH`. + + :exclamation: It is recommended to add only one version of the TensorRT/cuDNN runtime libraries to the `PATH`. It is better not to copy the runtime libraries of TensorRT/cuDNN to the cuda directory in `C:\`. + +7. Install pycuda by `pip install pycuda` + +## Model Convert + +### ONNX Runtime Example + +The following describes how to use the prebuilt package to do model conversion based on the previous downloaded pth. + +After preparation work, the structure of the current working directory should be： + +``` +.. +|-- mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1 +|-- mmclassification +|-- mmdeploy +`-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth +``` + +Model conversion can be performed like below: + +```python +from mmdeploy.apis import torch2onnx +from mmdeploy.backend.sdk.export_info import export2SDK + +img = 'mmclassification/demo/demo.JPEG' +work_dir = 'work_dir/onnx/resnet' +save_file = 'end2end.onnx' +deploy_cfg = 'mmdeploy/configs/mmcls/classification_onnxruntime_dynamic.py' +model_cfg = 'mmclassification/configs/resnet/resnet18_8xb32_in1k.py' +model_checkpoint = 'resnet18_8xb32_in1k_20210831-fbbb1da6.pth' +device = 'cpu' + +# 1. convert model to onnx +torch2onnx(img, work_dir, save_file, deploy_cfg, model_cfg, + model_checkpoint, device) + +# 2. extract pipeline info for sdk use (dump-info) +export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint) +``` + +The structure of the converted model directory: + +```bash +.\work_dir\ +`-- onnx + `-- resnet + |-- deploy.json + |-- detail.json + |-- end2end.onnx + `-- pipeline.json +``` + +### TensorRT Example + +The following describes how to use the prebuilt package to do model conversion based on the previous downloaded pth. + +After installation of mmdeploy-tensorrt prebuilt package, the structure of the current working directory should be： + +``` +.. +|-- mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0 +|-- mmclassification +|-- mmdeploy +`-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth +``` + +Model conversion can be performed like below: + +```python +from mmdeploy.apis import torch2onnx +from mmdeploy.apis.tensorrt import onnx2tensorrt +from mmdeploy.backend.sdk.export_info import export2SDK +import os + +img = 'mmclassification/demo/demo.JPEG' +work_dir = 'work_dir/trt/resnet' +save_file = 'end2end.onnx' +deploy_cfg = 'mmdeploy/configs/mmcls/classification_tensorrt_static-224x224.py' +model_cfg = 'mmclassification/configs/resnet/resnet18_8xb32_in1k.py' +model_checkpoint = 'resnet18_8xb32_in1k_20210831-fbbb1da6.pth' +device = 'cpu' + +# 1. convert model to IR(onnx) +torch2onnx(img, work_dir, save_file, deploy_cfg, model_cfg, + model_checkpoint, device) + +# 2. convert IR to tensorrt +onnx_model = os.path.join(work_dir, save_file) +save_file = 'end2end.engine' +model_id = 0 +device = 'cuda' +onnx2tensorrt(work_dir, save_file, model_id, deploy_cfg, onnx_model, device) + +# 3. extract pipeline info for sdk use (dump-info) +export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint) +``` + +The structure of the converted model directory: + +``` +.\work_dir\ +`-- trt + `-- resnet + |-- deploy.json + |-- detail.json + |-- end2end.engine + |-- end2end.onnx + `-- pipeline.json +``` + +## Model Inference + +You can obtain two model folders after model conversion. + +``` +.\work_dir\onnx\resnet +.\work_dir\trt\resnet +``` + +The structure of current working directory： + +``` +. +|-- mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0 +|-- mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1 +|-- mmclassification +|-- mmdeploy +|-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth +`-- work_dir +``` + +### Backend Inference + +:exclamation: It should be emphasized that `inference_model` is not for deployment, but shields the difference of backend inference api(`TensorRT`, `ONNX Runtime` etc.). The main purpose of this api is to check whether the converted model can be inferred normally. + +#### ONNXRuntime + +```python +from mmdeploy.apis import inference_model + +model_cfg = 'mmclassification/configs/resnet/resnet18_8xb32_in1k.py' +deploy_cfg = 'mmdeploy/configs/mmcls/classification_onnxruntime_dynamic.py' +backend_files = ['work_dir/onnx/resnet/end2end.onnx'] +img = 'mmclassification/demo/demo.JPEG' +device = 'cpu' +result = inference_model(model_cfg, deploy_cfg, backend_files, img, device) +``` + +#### TensorRT + +```python +from mmdeploy.apis import inference_model + +model_cfg = 'mmclassification/configs/resnet/resnet18_8xb32_in1k.py' +deploy_cfg = 'mmdeploy/configs/mmcls/classification_tensorrt_static-224x224.py' +backend_files = ['work_dir/trt/resnet/end2end.engine'] +img = 'mmclassification/demo/demo.JPEG' +device = 'cuda' +result = inference_model(model_cfg, deploy_cfg, backend_files, img, device) +``` + +### Python SDK + +The following describes how to use the SDK's Python API for inference + +#### ONNXRuntime + +```bash +python .\mmdeploy\demo\python\image_classification.py .\work_dir\onnx\resnet\ .\mmclassification\demo\demo.JPEG +``` + +#### TensorRT + +``` + python .\mmdeploy\demo\python\image_classification.py .\work_dir\trt\resnet\ .\mmclassification\demo\demo.JPEG --device-name cuda +``` + +### C SDK + +The following describes how to use the SDK's C API for inference + +#### ONNXRuntime + +1. Build examples + + Under `mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\example` directory + + ``` + // Path should be modified according to the actual location + mkdir build + cd build + cmake .. -A x64 -T v142 ` + -DOpenCV_DIR=C:\Deps\opencv\build\x64\vc15\lib ` + -DMMDeploy_DIR=C:\workspace\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\lib\cmake\MMDeploy ` + -DONNXRUNTIME_DIR=C:\Deps\onnxruntime\onnxruntime-win-gpu-x64-1.8.1 + + cmake --build . --config Release + ``` + +2. Add environment variables or copy the runtime libraries to the same level directory of exe + + :point_right: The purpose is to make the exe find the relevant dll + + If choose to add environment variables, add the runtime libraries path of `mmdeploy` (`mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\bin`) to the `PATH`. + + If choose to copy the dynamic libraries, copy the dll in the bin directory to the same level directory of the just compiled exe (build/Release). + +3. Inference： + + It is recommended to use `CMD` here. + + Under `mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\\sdk\\example\\build\\Release` directory： + + ``` + .\image_classification.exe cpu C:\workspace\work_dir\onnx\resnet\ C:\workspace\mmclassification\demo\demo.JPEG + ``` + +#### TensorRT + +1. Build examples + + Under `mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\\sdk\\example` directory + + ``` + // Path should be modified according to the actual location + mkdir build + cd build + cmake .. -A x64 -T v142 ` + -DOpenCV_DIR=C:\Deps\opencv\build\x64\vc15\lib ` + -DMMDeploy_DIR=C:\workspace\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8 2.3.0\sdk\lib\cmake\MMDeploy ` + -DTENSORRT_DIR=C:\Deps\tensorrt\TensorRT-8.2.3.0 ` + -DCUDNN_DIR=C:\Deps\cudnn\8.2.1 + cmake --build . --config Release + ``` + +2. Add environment variables or copy the runtime libraries to the same level directory of exe + + :point_right: The purpose is to make the exe find the relevant dll + + If choose to add environment variables, add the runtime libraries path of `mmdeploy` (`mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\bin`) to the `PATH`. + + If choose to copy the dynamic libraries, copy the dll in the bin directory to the same level directory of the just compiled exe (build/Release). + +3. Inference + + It is recommended to use `CMD` here. + + Under `mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\\sdk\\example\\build\\Release` directory + + ``` + .\image_classification.exe cuda C:\workspace\work_dir\trt\resnet C:\workspace\mmclassification\demo\demo.JPEG + ``` + +## Troubleshooting + +If you encounter problems, please refer to [FAQ](../faq.md) diff --git a/docs/en/faq.md b/docs/en/faq.md index dec0174d3c..1eea9b08e3 100644 --- a/docs/en/faq.md +++ b/docs/en/faq.md @@ -50,6 +50,40 @@ print(torch.__file__) ``` +- enable_language(CUDA) error + + ``` + -- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.19044. + -- Found CUDA: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.1 (found version "11.1") + CMake Error at C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCompilerId.cmake:491 (message): + No CUDA toolset found. + Call Stack (most recent call first): + C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCompilerId.cmake:6 (CMAKE_DETERMINE_COMPILER_ID_BUILD) + C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCompilerId.cmake:59 (__determine_compiler_id_test) + C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCUDACompiler.cmake:339 (CMAKE_DETERMINE_COMPILER_ID) + C:/workspace/mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0/sdk/lib/cmake/MMDeploy/MMDeployConfig.cmake:27 (enable_language) + CMakeLists.txt:5 (find_package) + ``` + + **Cause：** CUDA Toolkit 11.1 was installed before Visual Studio, so the VS plugin was not installed. Or the version of VS is too new, so that the installation of the VS plugin is skipped during the installation of the CUDA Toolkit + + **Solution：** This problem can be solved by manually copying the four files in `C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\extras\visual_studio_integration\MSBuildExtensions` to `C:\Software\Microsoft Visual Studio\2022\Community\Msbuild\Microsoft\VC\v170\BuildCustomizations` The specific path should be changed according to the actual situation. + +### ONNX Runtime + +- Under Windows system, when visualizing model inference result failed with the following error: + ``` + onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Failed to load library, error code: 193 + ``` + **Cause：** In latest Windows systems, there are two `onnxruntime.dll` under the system path, and they will be loaded first, causing conflicts. + ``` + C:\Windows\SysWOW64\onnxruntime.dll + C:\Windows\System32\onnxruntime.dll + ``` + **Solution：** Choose one of the following two options + 1. Copy the dll in the lib directory of the downloaded onnxruntime to the directory where mmdeploy_onnxruntime_ops.dll locates (It is recommended to use Everything to search the ops dll) + 2. Rename the two dlls in the system path so that they cannot be loaded. + ### Pip - pip installed package but could not `import` them. diff --git a/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md b/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md new file mode 100644 index 0000000000..3bef241310 --- /dev/null +++ b/docs/zh_cn/02-how-to-run/prebuilt_package_windows.md @@ -0,0 +1,400 @@ +# Win10 下预编译包的使用 + +- [Win10 下预编译包的使用](#win10-下预编译包的使用) + - [准备工作](#准备工作) + - [ONNX Runtime](#onnx-runtime) + - [TensorRT](#tensorrt) + - [模型转换](#模型转换) + - [ONNX Runtime Example](#onnx-runtime-example) + - [TensorRT Example](#tensorrt-example) + - [模型推理](#模型推理) + - [Backend Inference](#backend-inference) + - [ONNXRuntime](#onnxruntime) + - [TensorRT](#tensorrt-1) + - [Python SDK](#python-sdk) + - [ONNXRuntime](#onnxruntime-1) + - [TensorRT](#tensorrt-2) + - [C SDK](#c-sdk) + - [ONNXRuntime](#onnxruntime-2) + - [TensorRT](#tensorrt-3) + - [可能遇到的问题](#可能遇到的问题) + +______________________________________________________________________ + +目前，`MMDeploy`在`Windows`平台下提供`TensorRT`以及`ONNX Runtime`两种预编译包，可以从[Releases](https://github.com/open-mmlab/mmdeploy/releases)获取。 + +本篇教程以`mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1.zip`和`mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0.zip`为例，展示预编译包的使用方法。 + +为了方便使用者快速上手，本教程以分类模型(mmclassification)为例，展示两种预编译包的使用方法。 + +预编译包的目录结构如下，其中`dist`文件夹为模型转换相关内容，`sdk`文件夹为模型推理相关内容。 + +``` +. +|-- dist +`-- sdk + |-- bin + |-- example + |-- include + |-- lib + `-- python +``` + +## 准备工作 + +使用预编译包来进行`模型转换`以及`模型推理`，除了预编译包的中的内容外，还需要安装一些第三方依赖库，下面分别介绍以`ONNX Runtime`、`TensorRT`为推理后端所要进行的准备工作。 + +两种推理后端环境准备工作中，其中一些操作是共有的，下面先介绍这些共有的操作，再分别介绍各自特有的操作。 + +首先新建一个工作目录workspace + +1. 请按照[get_started](../get_started.md)文档，准备虚拟环境，安装pytorch、torchvision、mmcv-full。若要使用SDK的C接口，需要安装vs2019+, OpenCV。 + + :point_right: 这里建议使用`pip`而不是`conda`安装pytorch、torchvision + +2. 克隆mmdeploy仓库 + + ```bash + git clone https://github.com/open-mmlab/mmdeploy.git + ``` + + :point_right: 这里主要为了使用configs文件，所以没有加`--recursive`来下载submodule，也不需要编译`mmdeploy` + +3. 安装mmclassification + + ```bash + git clone https://github.com/open-mmlab/mmclassification.git + cd mmclassification + pip install -e . + ``` + +4. 准备一个PyTorch的模型文件当作我们的示例 + + 这里选择了[resnet18_8xb32_in1k_20210831-fbbb1da6.pth](https://download.openmmlab.com/mmclassification/v0/resnet/resnet18_8xb32_in1k_20210831-fbbb1da6.pth)，对应的训练config为[resnet18_8xb32_in1k.py](https://github.com/open-mmlab/mmclassification/blob/master/configs/resnet/resnet18_8xb32_in1k.py) + +做好以上工作后，当前工作目录的结构应为： + +``` +. +|-- mmclassification +|-- mmdeploy +|-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth +``` + +### ONNX Runtime + +本节介绍`mmdeploy`使用`ONNX Runtime`推理所特有的环境准备工作 + +5. 安装`mmdeploy`（模型转换）以及`mmdeploy_python`（模型推理Python API）的预编译包 + + ```bash + # 先下载 mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1.zip + pip install .\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\dist\mmdeploy-0.6.0-py38-none-win_amd64.whl + pip install .\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\python\mmdeploy_python-0.6.0-cp38-none-win_amd64.whl + ``` + + :point_right: 如果之前安装过，需要先卸载后再安装。 + +6. 安装onnxruntime package + + ``` + pip install onnxruntime==1.8.1 + ``` + +7. 下载[`onnxruntime`](https://github.com/microsoft/onnxruntime/releases/tag/v1.8.1)，添加环境变量 + + 将onnxruntime的lib目录添加到PATH里面，如图所示，具体的路径根据个人情况更改。 + + ![sys-path](https://user-images.githubusercontent.com/16019484/181463801-1d7814a8-b256-46e9-86f2-c08de0bc150b.png) + :exclamation: 重启powershell让环境变量生效，可以通过 echo $env:PATH 来检查是否设置成功。 + +### TensorRT + +本节介绍`mmdeploy`使用`TensorRT`推理所特有的环境准备工作 + +5. 安装`mmdeploy`（模型转换）以及`mmdeploy_python`（模型推理Python API）的预编译包 + + ```bash + # 先下载 mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0.zip + pip install .\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\dist\mmdeploy-0.6.0-py38-none-win_amd64.whl + pip install .\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\python\mmdeploy_python-0.6.0-cp38-none-win_amd64.whl + ``` + + :point_right: 如果之前安装过，需要先卸载后再安装 + +6. 安装CUDA相关内容，并设置环境变量 + + - CUDA Toolkit 11.1 + - TensorRT 8.2.3.0 (python包 + 环境变量) + - cuDNN 8.2.1.0 + + 其中CUDA的环境变量在安装CUDA Toolkit后会自动添加，TensorRT以及cuDNN解压后需要自行添加运行库的路径到PATH，可参考onnxruntime的设置图例 + + :exclamation: 重启powershell让环境变量生效，可以通过 echo $env:PATH 来检查是否设置成功 + + :exclamation: 建议只添加一个版本的TensorRT的lib到PATH里面。不建议拷贝TensorRT的dll到C盘的cuda目录，在某些情况下，这样可以暴露dll的版本问题 + +7. 安装pycuda `pip install pycuda` + +## 模型转换 + +### ONNX Runtime Example + +下面介绍根据之前下载的ckpt来展示如果使用`mmdeploy`预编译包来进行模型转换 + +经过之前的准备工作，当前的工作目录结构应该为： + +``` +.. +|-- mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1 +|-- mmclassification +|-- mmdeploy +`-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth +``` + +python 转换代码 + +```python +from mmdeploy.apis import torch2onnx +from mmdeploy.backend.sdk.export_info import export2SDK + +img = 'mmclassification/demo/demo.JPEG' +work_dir = 'work_dir/onnx/resnet' +save_file = 'end2end.onnx' +deploy_cfg = 'mmdeploy/configs/mmcls/classification_onnxruntime_dynamic.py' +model_cfg = 'mmclassification/configs/resnet/resnet18_8xb32_in1k.py' +model_checkpoint = 'resnet18_8xb32_in1k_20210831-fbbb1da6.pth' +device = 'cpu' + +# 1. convert model to onnx +torch2onnx(img, work_dir, save_file, deploy_cfg, model_cfg, + model_checkpoint, device) + +# 2. extract pipeline info for sdk use (dump-info) +export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint) +``` + +转换后的模型目录结构应该为： + +```bash +.\work_dir\ +`-- onnx + `-- resnet + |-- deploy.json + |-- detail.json + |-- end2end.onnx + `-- pipeline.json +``` + +### TensorRT Example + +下面根据之前下载的ckpt来展示如果使用mmdeploy预编译包来进行模型转换 + +经过之前的准备工作，当前的工作目录结构应该为： + +``` +.. +|-- mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0 +|-- mmclassification +|-- mmdeploy +`-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth +``` + +python 转换代码 + +```python +from mmdeploy.apis import torch2onnx +from mmdeploy.apis.tensorrt import onnx2tensorrt +from mmdeploy.backend.sdk.export_info import export2SDK +import os + +img = 'mmclassification/demo/demo.JPEG' +work_dir = 'work_dir/trt/resnet' +save_file = 'end2end.onnx' +deploy_cfg = 'mmdeploy/configs/mmcls/classification_tensorrt_static-224x224.py' +model_cfg = 'mmclassification/configs/resnet/resnet18_8xb32_in1k.py' +model_checkpoint = 'resnet18_8xb32_in1k_20210831-fbbb1da6.pth' +device = 'cpu' + +# 1. convert model to IR(onnx) +torch2onnx(img, work_dir, save_file, deploy_cfg, model_cfg, + model_checkpoint, device) + +# 2. convert IR to tensorrt +onnx_model = os.path.join(work_dir, save_file) +save_file = 'end2end.engine' +model_id = 0 +device = 'cuda' +onnx2tensorrt(work_dir, save_file, model_id, deploy_cfg, onnx_model, device) + +# 3. extract pipeline info for sdk use (dump-info) +export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint) +``` + +转换后的模型目录结构应该为： + +``` +.\work_dir\ +`-- trt + `-- resnet + |-- deploy.json + |-- detail.json + |-- end2end.engine + |-- end2end.onnx + `-- pipeline.json +``` + +## 模型推理 + +以下内容假定已完成了上述模型转换的两个Example，并得到了上述模型转换后的两个文件夹其中之一或者全部： + +``` +.\work_dir\onnx\resnet +.\work_dir\trt\resnet +``` + +当前的工作目录应为： + +``` +. +|-- mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0 +|-- mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1 +|-- mmclassification +|-- mmdeploy +|-- resnet18_8xb32_in1k_20210831-fbbb1da6.pth +`-- work_dir +``` + +### Backend Inference + +:exclamation: 需要强调的一点是，这个接口不是为了做部署的，而是屏蔽了推理后端接口的，用来检验转换的模型是否可以正常推理的。 + +#### ONNXRuntime + +Python 代码 + +```python +from mmdeploy.apis import inference_model + +model_cfg = 'mmclassification/configs/resnet/resnet18_8xb32_in1k.py' +deploy_cfg = 'mmdeploy/configs/mmcls/classification_onnxruntime_dynamic.py' +backend_files = ['work_dir/onnx/resnet/end2end.onnx'] +img = 'mmclassification/demo/demo.JPEG' +device = 'cpu' +result = inference_model(model_cfg, deploy_cfg, backend_files, img, device) +``` + +#### TensorRT + +Python 代码 + +```python +from mmdeploy.apis import inference_model + +model_cfg = 'mmclassification/configs/resnet/resnet18_8xb32_in1k.py' +deploy_cfg = 'mmdeploy/configs/mmcls/classification_tensorrt_static-224x224.py' +backend_files = ['work_dir/trt/resnet/end2end.engine'] +img = 'mmclassification/demo/demo.JPEG' +device = 'cuda' +result = inference_model(model_cfg, deploy_cfg, backend_files, img, device) +``` + +### Python SDK + +这里介绍如何使用SDK的Python API进行推理 + +#### ONNXRuntime + +推理代码 + +```bash +python .\mmdeploy\demo\python\image_classification.py .\work_dir\onnx\resnet\ .\mmclassification\demo\demo.JPEG +``` + +#### TensorRT + +推理代码 + +``` + python .\mmdeploy\demo\python\image_classification.py .\work_dir\trt\resnet\ .\mmclassification\demo\demo.JPEG --device-name cuda +``` + +### C SDK + +这里介绍如何使用SDK的C API进行推理 + +#### ONNXRuntime + +1. 编译 examples + + 在`mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\example`目录下 + + ``` + // 部分路径根据实际位置进行修改 + mkdir build + cd build + cmake .. -A x64 -T v142 ` + -DOpenCV_DIR=C:\Deps\opencv\build\x64\vc15\lib ` + -DMMDeploy_DIR=C:\workspace\mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\lib\cmake\MMDeploy ` + -DONNXRUNTIME_DIR=C:\Deps\onnxruntime\onnxruntime-win-gpu-x64-1.8.1 + + cmake --build . --config Release + ``` + +2. 添加环境变量或拷贝动态库到exe同级目录 + + :point_right: 目的是使exe运行时可以正确找到相关dll + + 若选择添加环境变量，则将`mmdeploy`的运行时库路径（`mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\sdk\bin`）添加到PATH，可参考onnxruntime的添加过程。 + + 若选择拷贝动态库，而将bin目录中的dll拷贝到刚才编译出的exe(build/Release)的同级目录下。 + +3. 推理： + + 这里建议使用cmd，这样如果exe运行时如果找不到相关的dll的话会有弹窗 + + 在mmdeploy-0.6.0-windows-amd64-onnxruntime1.8.1\\sdk\\example\\build\\Release目录下： + + ``` + .\image_classification.exe cpu C:\workspace\work_dir\onnx\resnet\ C:\workspace\mmclassification\demo\demo.JPEG + ``` + +#### TensorRT + +1. 编译 examples + + 在mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\\sdk\\example目录下 + + ``` + // 部分路径根据所在硬盘的位置进行修改 + mkdir build + cd build + cmake .. -A x64 -T v142 ` + -DOpenCV_DIR=C:\Deps\opencv\build\x64\vc15\lib ` + -DMMDeploy_DIR=C:\workspace\mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8 2.3.0\sdk\lib\cmake\MMDeploy ` + -DTENSORRT_DIR=C:\Deps\tensorrt\TensorRT-8.2.3.0 ` + -DCUDNN_DIR=C:\Deps\cudnn\8.2.1 + cmake --build . --config Release + ``` + +2. 添加环境变量或拷贝动态库到exe同级目录 + + :point_right: 目的是使exe运行时可以正确找到相关dll + + 若选择添加环境变量，则将`mmdeploy`的运行时库路径（`mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\sdk\bin`）添加到PATH，可参考onnxruntime的添加过程。 + + 若选择拷贝动态库，而将bin目录中的dll拷贝到刚才编译出的exe(build/Release)的同级目录下。 + +3. 推理 + + 这里建议使用cmd，这样如果exe运行时如果找不到相关的dll的话会有弹窗 + + 在mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0\\sdk\\example\\build\\Release目录下： + + ``` + .\image_classification.exe cuda C:\workspace\work_dir\trt\resnet C:\workspace\mmclassification\demo\demo.JPEG + ``` + +## 可能遇到的问题 + +如遇到问题，可参考[FAQ](../faq.md) diff --git a/docs/zh_cn/faq.md b/docs/zh_cn/faq.md index 8b227cc9fc..67615b1d14 100644 --- a/docs/zh_cn/faq.md +++ b/docs/zh_cn/faq.md @@ -50,6 +50,40 @@ print(torch.__file__) ``` +- 编译时enable_language(CUDA) 报错 + + ``` + -- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.19044. + -- Found CUDA: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.1 (found version "11.1") + CMake Error at C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCompilerId.cmake:491 (message): + No CUDA toolset found. + Call Stack (most recent call first): + C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCompilerId.cmake:6 (CMAKE_DETERMINE_COMPILER_ID_BUILD) + C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCompilerId.cmake:59 (__determine_compiler_id_test) + C:/Software/cmake/cmake-3.23.1-windows-x86_64/share/cmake-3.23/Modules/CMakeDetermineCUDACompiler.cmake:339 (CMAKE_DETERMINE_COMPILER_ID) + C:/workspace/mmdeploy-0.6.0-windows-amd64-cuda11.1-tensorrt8.2.3.0/sdk/lib/cmake/MMDeploy/MMDeployConfig.cmake:27 (enable_language) + CMakeLists.txt:5 (find_package) + ``` + + **原因：** CUDA Toolkit 11.1安装在Visual Studio之前，造成VS的插件没有安装。或者VS的版本过新，使得CUDA Toolkit的安装的时候跳过了VS插件的安装 + + **解决方法：** 可以通过手工拷贝插件的方式来解决这个问题。比如将`C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\extras\visual_studio_integration\MSBuildExtensions`中的四个文件拷贝到`C:\Software\Microsoft Visual Studio\2022\Community\Msbuild\Microsoft\VC\v170\BuildCustomizations` 目录下。具体路径根据实际情况进行更改。 + +### ONNX Runtime + +- Windows系统下，转模型可视化时以及SDK推理时遇到 + ``` + onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Failed to load library, error code: 193 + ``` + **原因：** 在较新的windows系统中，系统路径下下有两个`onnxruntime.dll`，且会优先加载，造成冲突。 + ``` + C:\Windows\SysWOW64\onnxruntime.dll + C:\Windows\System32\onnxruntime.dll + ``` + **解决方法：** 以下两个方案任选其一 + 1. 将下载的onnxruntime中的lib目录下的dll拷贝到mmdeploy_onnxruntime_ops.dll的同级目录（推荐使用Everything 进行查找） + 2. 将系统路径下的这两个dll改名，使其加载不到，可能涉及到修改文件权限的操作 + ### Pip - pip installed package but could not `import` them.