Paddle如何支持Resnet半精度推理？

[xiaoyewww]

bug描述 Describe the Bug

请问一下paddle如何支持resnet半精度推理，我在代码中下载resnet相关模型及其权重后，加载模型转成fp16，想实现paddle原生半精度推理：

import os
import numpy as np
import paddle


if __name__ == "__main__":
    # download resnet model
    if not os.path.exists("ResNet50_infer"):
        os.system("wget https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/inference/ResNet50_infer.tar && tar -xf ResNet50_infer.tar && rm -rf ResNet50_infer.tar")

    # generate fp16 model
    path = "ResNet50_infer/inference"
    # paddle.set_device("gpu")
    model = paddle.jit.load(path)
    model.float16()
    model.eval()

    np.random.seed(10)
    input_img = np.random.rand(1, 3, 224, 224).astype("float16")

    # resnet50 cannot be inferenced by half?
    paddle_input = paddle.to_tensor(input_img, dtype="float16")
    paddle_output = model(paddle_input)

上述代码报错信息：

Traceback (most recent call last):
  File "/wuzp/Paddle2ONNX/tests/test_resnet_fp16.py", line 37, in <module>
    paddle_output = model(paddle_input)
  File "/root/miniconda3/envs/paddle_onnx/lib/python3.9/site-packages/paddle/nn/layer/layers.py", line 1429, in __call__
    return self.forward(*inputs, **kwargs)
  File "/root/miniconda3/envs/paddle_onnx/lib/python3.9/site-packages/paddle/jit/translated_layer.py", line 1477, in __i_m_p_l__
    return _run_dygraph(self, input, program_holder)
  File "/root/miniconda3/envs/paddle_onnx/lib/python3.9/site-packages/paddle/jit/translated_layer.py", line 1004, in _run_dygraph
    _legacy_C_ops.run_program(
ValueError: (InvalidArgument) Scale input should be of float type
  [Hint: Expected bn_param_type == framework::TransToProtoVarType( ctx.Input<phi::DenseTensor>("Scale")->dtype()), but received bn_param_type:5 != framework::TransToProtoVarType( ctx.Input<phi::DenseTensor>("Scale")->dtype()):4.] (at ../paddle/fluid/operators/batch_norm_op.cc:198)
  [operator < batch_norm > error]  [operator < run_program > error]

其他补充信息 Additional Supplementary Information

paddlepaddle-gpu 2.6.1.post117

[vivienfanghuagood]

你好，关于fp16的resnet推理，请参考这个demo：https://github.com/PaddlePaddle/Paddle-Inference-Demo/tree/master/c%2B%2B/gpu/gpu_fp16

[Zheng-Bicheng]

你好，关于fp16的resnet推理，请参考这个demo：https://github.com/PaddlePaddle/Paddle-Inference-Demo/tree/master/c%2B%2B/gpu/gpu_fp16

大佬，只能用C++做推理吗？没有Python的API接口来进行推理吗？

[xiaoyewww]

@vivienfanghuagood 大佬，这个问题还能解答一下吗，我发现这里没法跑fp16的原因应该是type不统一，python上详细的接口提供吗？

[vivienfanghuagood]

你好，关于fp16的resnet推理，请参考这个demo：https://github.com/PaddlePaddle/Paddle-Inference-Demo/tree/master/c%2B%2B/gpu/gpu_fp16

大佬，只能用C++做推理吗？没有Python的API接口来进行推理吗？

有的，python端的demo: https://github.com/PaddlePaddle/Paddle-Inference-Demo/blob/master/python/gpu/resnet50/infer_resnet.py#L34

[vivienfanghuagood]

https://www.paddlepaddle.org.cn/inference/v2.6/guides/nv_gpu_infer/gpu_mixed_precision.html#paddle-gpu 也可以参考这个文档

[xiaoyewww]

https://www.paddlepaddle.org.cn/inference/v2.6/guides/nv_gpu_infer/gpu_mixed_precision.html#paddle-gpu 也可以参考这个文档

@vivienfanghuagood 感谢，可以转换出fp16的模型，但又有一个问题，我没法通过paddle.jit.load来加载这个模型吗：

# download resnet model
if not os.path.exists("ResNet50_infer"):
    os.system("wget https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/inference/ResNet50_infer.tar && tar -xf ResNet50_infer.tar && rm -rf ResNet50_infer.tar")

# generate fp16 model
path = "ResNet50_infer"
src_model = os.path.join(path,"inference.pdmodel")
src_params = os.path.join(path,"inference.pdiparams")
dst_model = os.path.join(path,"inference_fp16.pdmodel")
dst_params = os.path.join(path,"inference_fp16.pdiparams")
black_list = set()

convert_to_mixed_precision(
    src_model,     # fp32模型文件路径
    src_params,    # fp32权重文件路径
    dst_model,     # 混合精度模型文件保存路径
    dst_params,    # 混合精度权重文件保存路径
    PrecisionType.Half, # 转换精度，如 PrecisionType.Half
    PlaceType.GPU,      # 后端，如 PlaceType.GPU
    False,               # 保留输入输出精度信息，若为 True 则输入输出保留为 fp32 类型，否则转为 precision 类型
    black_list          # 黑名单列表，哪些 op 不需要进行精度类型转换
)

# paddle.set_device("gpu")
path_fp16 = os.path.join(path, "inference_fp16")
print(f"path_fp16: {path_fp16}")
model = paddle.jit.load(path_fp16)
model.eval()

报错如下：

/root/miniconda3/envs/paddle_onnx/lib/python3.9/site-packages/decorator.py:232: in fun
    return caller(func, *(extras + args), **kw)
/root/miniconda3/envs/paddle_onnx/lib/python3.9/site-packages/paddle/base/wrapped_decorator.py:26: in __impl__
    return wrapped_func(*args, **kwargs)
/root/miniconda3/envs/paddle_onnx/lib/python3.9/site-packages/paddle/base/framework.py:593: in __impl__
    return func(*args, **kwargs)
/root/miniconda3/envs/paddle_onnx/lib/python3.9/site-packages/paddle/jit/api.py:1584: in load
    return TranslatedLayer._construct(model_path, config)
/root/miniconda3/envs/paddle_onnx/lib/python3.9/site-packages/decorator.py:232: in fun
    return caller(func, *(extras + args), **kw)
/root/miniconda3/envs/paddle_onnx/lib/python3.9/site-packages/paddle/base/wrapped_decorator.py:26: in __impl__
    return wrapped_func(*args, **kwargs)
/root/miniconda3/envs/paddle_onnx/lib/python3.9/site-packages/paddle/base/framework.py:593: in __impl__
    return func(*args, **kwargs)
/root/miniconda3/envs/paddle_onnx/lib/python3.9/site-packages/paddle/jit/translated_layer.py:1444: in _construct
    persistable_vars = _construct_params_and_buffers(
/root/miniconda3/envs/paddle_onnx/lib/python3.9/site-packages/paddle/jit/translated_layer.py:874: in _construct_params_and_buffers
    var_dict = _load_persistable_vars_by_program(
/root/miniconda3/envs/paddle_onnx/lib/python3.9/site-packages/paddle/jit/translated_layer.py:714: in _load_persistable_vars_by_program
    all_var_names = _get_all_var_names(program_holder.train_program)
/root/miniconda3/envs/paddle_onnx/lib/python3.9/site-packages/paddle/jit/dy2static/partial_program.py:110: in __get__
    val = self.function(instance)
/root/miniconda3/envs/paddle_onnx/lib/python3.9/site-packages/paddle/jit/translated_layer.py:389: in train_program
    return self._append_backward_desc(self._infer_program_desc)
/root/miniconda3/envs/paddle_onnx/lib/python3.9/site-packages/decorator.py:232: in fun
    return caller(func, *(extras + args), **kw)
/root/miniconda3/envs/paddle_onnx/lib/python3.9/site-packages/paddle/base/wrapped_decorator.py:26: in __impl__
    return wrapped_func(*args, **kwargs)
/root/miniconda3/envs/paddle_onnx/lib/python3.9/site-packages/paddle/base/dygraph/base.py:68: in __impl__
    return func(*args, **kwargs)
/root/miniconda3/envs/paddle_onnx/lib/python3.9/site-packages/paddle/jit/translated_layer.py:604: in _append_backward_desc
    grad_info_map = backward.calc_gradient_helper(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

targets = [], inputs = [], target_gradients = [], no_grad_set = None

    def calc_gradient_helper(
        targets, inputs, target_gradients=None, no_grad_set=None
    ):
        '''
        Calculate gradient and return grad_info_map
        '''
        targets = _as_list(targets)
        inputs = _as_list(inputs)
        target_gradients = _as_list(target_gradients)
    
>       block = targets[0].block
E       IndexError: list index out of range

/root/miniconda3/envs/paddle_onnx/lib/python3.9/site-packages/paddle/base/backward.py:2472: IndexError

看上去没反向信息就不能加载了

[xiaoyewww]

@vivienfanghuagood 感谢，已经没有问题了。