update deployment guide (#1551)

* update deployment guide * update * fix lint error * update * fix typo * Update docs/zh_cn/user_guides/deploy.md * Update docs/en/user_guides/deploy.md * Update docs/en/user_guides/deploy.md * Update docs/zh_cn/user_guides/deploy.md Co-authored-by: Yanhong Zeng <[email protected]>
open-mmlab · Dec 26, 2022 · 8746f8d · 8746f8d
1 parent c1faa1e
commit 8746f8d
Show file tree

Hide file tree

Showing 2 changed files with 311 additions and 3 deletions.
diff --git a/docs/en/user_guides/deploy.md b/docs/en/user_guides/deploy.md
@@ -1,4 +1,158 @@
 # Tutorial 8: Deploy models in MMEditing
 
-[MMDeploy](https://github.com/open-mmlab/mmdeploy) is an open-source deep learning model deployment toolset.
-MMDeploy supports deploying models in MMEditing. Please refer to [MMDeploy](https://github.com/open-mmlab/mmdeploy) for more information.
+The deployment of OpenMMLab codebases, including MMClassification, MMDetection, MMEditing and so on are supported by [MMDeploy](https://github.com/open-mmlab/mmdeploy).
+The latest deployment guide for MMEditing can be found from [here](https://mmdeploy.readthedocs.io/en/1.x/04-supported-codebases/mmedit.html).
+
+This tutorial is organized as follows:
+
+- [Installation](#installation)
+- [Convert model](#convert-model)
+- [Model specification](#model-specification)
+- [Model inference](#model-inference)
+  - [Backend model inference](#backend-model-inference)
+  - [SDK model inference](#sdk-model-inference)
+- [Supported models](#supported-models)
+
+## Installation
+
+Please follow the [guide](../get_started/install.md) to install mmedit. And then install mmdeploy from source by following [this](https://mmdeploy.readthedocs.io/en/1.x/get_started.html#installation) guide.
+
+```{note}
+If you install mmdeploy prebuilt package, please also clone its repository by 'git clone https://github.com/open-mmlab/mmdeploy.git --depth=1' to get the deployment config files.
+```
+
+## Convert model
+
+Suppose mmediting and mmdeploy repositories are in the same directory, and the working directory is the root path of mmediting.
+
+Take [ESRGAN](../../../configs/esrgan/esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py) model as an example.
+You can download its checkpoint from [here](https://download.openmmlab.com/mmediting/restorers/esrgan/esrgan_psnr_x4c64b23g32_1x16_1000k_div2k_20200420-bf5c993c.pth), and then convert it to onnx model as follows:
+
+```python
+from mmdeploy.apis import torch2onnx
+from mmdeploy.backend.sdk.export_info import export2SDK
+
+img = 'tests/data/image/face/000001.png'
+work_dir = 'mmdeploy_models/mmedit/onnx'
+save_file = 'end2end.onnx'
+deploy_cfg = '../mmdeploy/configs/mmedit/super-resolution/super-resolution_onnxruntime_dynamic.py'
+model_cfg = 'configs/esrgan/esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py'
+model_checkpoint = 'esrgan_psnr_x4c64b23g32_1x16_1000k_div2k_20200420-bf5c993c.pth'
+device = 'cpu'
+
+# 1. convert model to onnx
+torch2onnx(img, work_dir, save_file, deploy_cfg, model_cfg,
+  model_checkpoint, device)
+
+# 2. extract pipeline info for inference by MMDeploy SDK
+export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint, device=device)
+```
+
+It is crucial to specify the correct deployment config during model conversion.MMDeploy has already provided builtin deployment config [files](https://github.com/open-mmlab/mmdeploy/tree/1.x/configs/mmedit) of all supported backends for mmedit, under which the config file path follows the pattern:
+
+```
+{task}/{task}_{backend}-{precision}_{static | dynamic}_{shape}.py
+```
+
+- **{task}:** task in mmedit.
+
+- **{backend}:** inference backend, such as onnxruntime, tensorrt, pplnn, ncnn, openvino, coreml etc.
+
+- **{precision}:** fp16, int8. When it's empty, it means fp32
+
+- **{static | dynamic}:** static shape or dynamic shape
+
+- **{shape}:** input shape or shape range of a model
+
+Therefore, in the above example, you can also convert `ESRGAN` to other backend models by changing the deployment config file, e.g., converting to tensorrt-fp16 model by `super-resolution_tensorrt-fp16_dynamic-32x32-512x512.py`.
+
+```{tip}
+When converting mmedit models to tensorrt models, --device should be set to "cuda"
+```
+
+## Model specification
+
+Before moving on to model inference chapter, let's know more about the converted model structure which is very important for model inference.
+
+The converted model locates in the working directory like `mmdeploy_models/mmedit/onnx` in the previous example. It includes:
+
+```
+mmdeploy_models/mmedit/onnx
+├── deploy.json
+├── detail.json
+├── end2end.onnx
+└── pipeline.json
+```
+
+in which,
+
+- **end2end.onnx**: backend model which can be inferred by ONNX Runtime
+- ***xxx*.json**: the necessary information for mmdeploy SDK
+
+The whole package **mmdeploy_models/mmedit/onnx** is defined as **mmdeploy SDK model**, i.e., **mmdeploy SDK model** includes both backend model and inference meta information.
+
+## Model inference
+
+### Backend model inference
+
+Take the previous converted `end2end.onnx` model as an example, you can use the following code to inference the model.
+
+```python
+from mmdeploy.apis.utils import build_task_processor
+from mmdeploy.utils import get_input_shape, load_config
+import torch
+
+deploy_cfg = '../mmdeploy/configs/mmedit/super-resolution/super-resolution_onnxruntime_dynamic.py'
+model_cfg = 'configs/esrgan/esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py'
+device = 'cpu'
+backend_model = ['mmdeploy_models/mmedit/onnx/end2end.onnx']
+image = 'tests/data/image/lq/baboon_x4.png'
+
+# read deploy_cfg and model_cfg
+deploy_cfg, model_cfg = load_config(deploy_cfg, model_cfg)
+
+# build task and backend model
+task_processor = build_task_processor(model_cfg, deploy_cfg, device)
+model = task_processor.build_backend_model(backend_model)
+
+# process input image
+input_shape = get_input_shape(deploy_cfg)
+model_inputs, _ = task_processor.create_input(image, input_shape)
+
+# do model inference
+with torch.no_grad():
+    result = model.test_step(model_inputs)
+
+# visualize results
+task_processor.visualize(
+    image=image,
+    model=model,
+    result=result[0],
+    window_name='visualize',
+    output_file='output_restorer.bmp')
+```
+
+### SDK model inference
+
+You can also perform SDK model inference like following,
+
+```python
+from mmdeploy_python import Restorer
+import cv2
+
+img = cv2.imread('tests/data/image/lq/baboon_x4.png')
+
+# create a predictor
+restorer = Restorer(model_path='mmdeploy_models/mmedit/onnx', device_name='cpu', device_id=0)
+# perform inference
+result = restorer(img)
+
+# visualize inference result
+cv2.imwrite('output_restorer.bmp', result)
+```
+
+Besides python API, MMDeploy SDK also provides other FFI (Foreign Function Interface), such as C, C++, C#, Java and so on. You can learn their usage from [demos](https://github.com/open-mmlab/mmdeploy/tree/1.x/demo).
+
+## Supported models
+
+Please refer to [here](https://mmdeploy.readthedocs.io/en/1.x/04-supported-codebases/mmedit.html#supported-models) for the supported model list.
diff --git a/docs/zh_cn/user_guides/deploy.md b/docs/zh_cn/user_guides/deploy.md
@@ -1 +1,155 @@
-# 教程 8：部署指南（待更新）
+# 教程 8：模型部署指南
+
+[MMDeploy](https://github.com/open-mmlab/mmdeploy) 是 OpenMMLab 的部署仓库，负责包括 MMClassification、MMDetection、MMEditing 等在内的各算法库的部署工作。
+你可以从[这里](https://mmdeploy.readthedocs.io/zh_CN/1.x/04-supported-codebases/mmedit.html)获取 MMDeploy 对 MMClassification 部署支持的最新文档。
+
+本文的结构如下：
+
+- [安装](#安装)
+- [模型转换](#模型转换)
+- [模型规范](#模型规范)
+- [模型推理](#模型推理)
+  - [后端模型推理](#后端模型推理)
+  - [SDK 模型推理](#sdk-模型推理)
+- [模型支持列表](#模型支持列表)
+
+## 安装
+
+请参考[此处](../get_started/install.md)安装 mmedit。然后，按照[说明](https://mmdeploy.readthedocs.io/zh_CN/1.x/get_started.html#mmdeploy)安装 mmdeploy。
+
+```{note}
+如果安装的是 mmdeploy 预编译包，那么也请通过 'git clone https://github.com/open-mmlab/mmdeploy.git --depth=1' 下载 mmdeploy 源码。因为它包含了部署时要用到的配置文件
+```
+
+## 模型转换
+
+假设在安装步骤中，mmediting 和 mmdeploy 代码库在同级目录下，并且当前的工作目录为 mmediting 的根目录，那么以 [ESRGAN](../../../configs/esrgan/esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py) 模型为例，你可以从[此处](https://download.openmmlab.com/mmediting/restorers/esrgan/esrgan_psnr_x4c64b23g32_1x16_1000k_div2k_20200420-bf5c993c.pth)下载对应的 checkpoint，并使用以下代码将之转换为 onnx 模型：
+
+```python
+from mmdeploy.apis import torch2onnx
+from mmdeploy.backend.sdk.export_info import export2SDK
+
+img = 'tests/data/image/face/000001.png'
+work_dir = 'mmdeploy_models/mmedit/onnx'
+save_file = 'end2end.onnx'
+deploy_cfg = '../mmdeploy/configs/mmedit/super-resolution/super-resolution_onnxruntime_dynamic.py'
+model_cfg = 'configs/esrgan/esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py'
+model_checkpoint = 'esrgan_psnr_x4c64b23g32_1x16_1000k_div2k_20200420-bf5c993c.pth'
+device = 'cpu'
+
+# 1. convert model to onnx
+torch2onnx(img, work_dir, save_file, deploy_cfg, model_cfg,
+  model_checkpoint, device)
+
+# 2. extract pipeline info for inference by MMDeploy SDK
+export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint, device=device)
+```
+
+转换的关键之一是使用正确的配置文件。项目中已内置了各后端部署[配置文件](https://github.com/open-mmlab/mmdeploy/tree/1.x/configs/mmedit)。
+文件的命名模式是：
+
+```
+{task}/{task}_{backend}-{precision}_{static | dynamic}_{shape}.py
+```
+
+其中：
+
+- **{task}:** mmedit 中的任务
+- **{backend}:** 推理后端名称。比如，onnxruntime、tensorrt、pplnn、ncnn、openvino、coreml 等等
+- **{precision}:** 推理精度。比如，fp16、int8。不填表示 fp32
+- **{static | dynamic}:** 动态、静态 shape
+- **{shape}:** 模型输入的 shape 或者 shape 范围
+
+在上例中，你也可以把 `ESRGAN` 转为其他后端模型。比如使用`super-resolution_tensorrt-fp16_dynamic-32x32-512x512.py`，把模型转为 tensorrt-fp16 模型。
+
+```{tip}
+当转 tensorrt 模型时, --device 需要被设置为 "cuda"
+```
+
+## 模型规范
+
+在使用转换后的模型进行推理之前，有必要了解转换结果的结构。 它存放在 `--work-dir` 指定的路路径下。
+
+上例中的`mmdeploy_models/mmedit/onnx`，结构如下：
+
+```
+mmdeploy_models/mmedit/onnx
+├── deploy.json
+├── detail.json
+├── end2end.onnx
+└── pipeline.json
+```
+
+重要的是：
+
+- **end2end.onnx**: 推理引擎文件。可用 ONNX Runtime 推理
+- ***xxx*.json**:  mmdeploy SDK 推理所需的 meta 信息
+
+整个文件夹被定义为**mmdeploy SDK model**。换言之，**mmdeploy SDK model**既包括推理引擎，也包括推理 meta 信息。
+
+## 模型推理
+
+### 后端模型推理
+
+以上述模型转换后的 `end2end.onnx` 为例，你可以使用如下代码进行推理：
+
+```python
+from mmdeploy.apis.utils import build_task_processor
+from mmdeploy.utils import get_input_shape, load_config
+import torch
+
+deploy_cfg = '../mmdeploy/configs/mmedit/super-resolution/super-resolution_onnxruntime_dynamic.py'
+model_cfg = 'configs/esrgan/esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py'
+device = 'cpu'
+backend_model = ['mmdeploy_models/mmedit/onnx/end2end.onnx']
+image = 'tests/data/image/lq/baboon_x4.png'
+
+# read deploy_cfg and model_cfg
+deploy_cfg, model_cfg = load_config(deploy_cfg, model_cfg)
+
+# build task and backend model
+task_processor = build_task_processor(model_cfg, deploy_cfg, device)
+model = task_processor.build_backend_model(backend_model)
+
+# process input image
+input_shape = get_input_shape(deploy_cfg)
+model_inputs, _ = task_processor.create_input(image, input_shape)
+
+# do model inference
+with torch.no_grad():
+    result = model.test_step(model_inputs)
+
+# visualize results
+task_processor.visualize(
+    image=image,
+    model=model,
+    result=result[0],
+    window_name='visualize',
+    output_file='output_restorer.bmp')
+```
+
+### SDK 模型推理
+
+你也可以参考如下代码，对 SDK model 进行推理：
+
+```python
+from mmdeploy_python import Restorer
+import cv2
+
+img = cv2.imread('tests/data/image/lq/baboon_x4.png')
+
+# create a predictor
+restorer = Restorer(model_path='mmdeploy_models/mmedit/onnx', device_name='cpu', device_id=0)
+# perform inference
+result = restorer(img)
+
+# visualize inference result
+cv2.imwrite('output_restorer.bmp', result)
+```
+
+除了python API，mmdeploy SDK 还提供了诸如 C、C++、C#、Java等多语言接口。
+你可以参考[样例](https://github.com/open-mmlab/mmdeploy/tree/1.x/demo)学习其他语言接口的使用方法。
+
+## 模型支持列表
+
+请参考[这里](https://mmdeploy.readthedocs.io/zh_CN/1.x/04-supported-codebases/mmedit.html#id7)