[BYOC] [TPAT] [TensorRT] Add the ability to automatically generate TensorRT plugins using TVM

[Civitasv]

TPAT: TVM Plugin Autogen Tool

Disclaimer: This PR is based on Tencent's TPAT.

Purpose: Tencent's TPAT should be used with their TVM fork: BlazerML-tvm, but they haven't synchronized it with the upstream for a long time, also some bugs are not resolved. In light of these issues, I decide to try integrating it to TVM.

Objective: The primary goal is to offer a clear and user-friendly API.

Architecture

Currently, only TensorRT is supported.

In essence, this solution is built upon the Template Engine (Jinja) in Python to create plugin templates for vendor-specific acceleration libraries. It then utilizes TVM for optimization and code generation targeting the respective platforms. The generated code is rendered and filled into the templates. Subsequently, platform-specific build commands are invoked to build the plugins, which ultimately serve as extensions for the corresponding vendor's acceleration library.

Inputs & Outputs

The entry of TPAT for TensorRT is as follows:

def pipeline(
    onnx_file: str, node_names: list[str], enable_tunning: bool, work_dir: str, output_onnx: str
) -> Tuple[str, list[str]]:

This entry point accepts an ONNX file, a list of nodes to be tuned, the log database location, and the output ONNX file path where the modified model will be stored.

After generating plugins for each node, the function returns the path of the output ONNX file along with a list of paths where the plugins are saved, facilitating subsequent loading.

TODO

• User should have the ability to change tunning option.
• Add benchmark section.
• Currently, the frontend is Relay, the tunning method is MetaSchedule, we should a flexible way to support Relax and other tunning method.
• Consider potential improvements to the API on the C++ side. Currently I use some global variables, then register global functions to get these variables, it feels like a hack to me, anyway, I'm not quite familiar with TVM's way to do it, so please give me some advice.
• Explore dynamic batch support, currently, only static batch is supported, the original repo supports it, but I think it's a little mess, I believe there exists a more elegant way to support this feature.
• Investigate the generation of QNN plugins for Qualcomm platforms.

Reference

• Introduction of Tencent's TPAT
• TPAT: TensorRT Plugin Autogen Tool - by Tencent @ TVMCon 2021

[tvm-bot]

Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from Reviewers by @-ing them in a comment.

• cc @billishyahao _{See #10317 for details}

_{Generated by tvm-bot}

[Civitasv]

cc @tqchen @Hzfengsy @FrozenGene

[Hzfengsy]

Thanks, @Civitasv for this great work! There are notable things:

1. This is the improvement based on Relay, not Relax, so it should be sent to main branch instead of the unity branch
2. It's an awesome and big feature, having an RFC (https://github.com/apache/tvm-rfcs) before PR would be good
3. This PR is a bit large to review, could it be separated into several small ones, together with a tracking issue after the RFC

[Civitasv]

This is the improvement based on Relay, not Relax, so it should be sent to main branch instead of the unity branch

The final goal is to support both Relay and Relax, but I agree currently it should be sent to main branch.

• It's an awesome and big feature, having an RFC (https://github.com/apache/tvm-rfcs) before PR would be good

Okay, I will write an RFC.

This PR is a bit large to review, could it be separated into several small ones, together with a tracking issue after the RFC

I will try to separate it.

[Civitasv]

I've already proposed an RFC. See apache/tvm-rfcs#103.

[buptqq]

I've already proposed an RFC. See apache/tvm-rfcs#103.
Hi, I am the author of TPAT. If you need any help, you can contact me through this email : [email protected]

[Civitasv]

I've already proposed an RFC. See apache/tvm-rfcs#103.
Hi, I am the author of TPAT. If you need any help, you can contact me through this email : [email protected]

@buptqq Thanks for your great work! It helps me a lot, If you are still working at this project, can you review the code? I've changed much.

[Civitasv]

I've improved the code, the workflow should be clear if you've read the RFC. 😄

[buptqq]

I've already proposed an RFC. See apache/tvm-rfcs#103.
Hi, I am the author of TPAT. If you need any help, you can contact me through this email : [email protected]

@buptqq Thanks for your great work! It helps me a lot, If you are still working at this project, can you review the code? I've changed much.

OK, I will review this code.