Replies: 4 comments 4 replies
-
@narendasan What are your thoughts on this ? |
Beta Was this translation helpful? Give feedback.
-
Can you explain the separate library here? |
Beta Was this translation helpful? Give feedback.
-
Ideally the user never needs to think about if IMO the idea should be that The main reason to have a separate .so is to support people doing separate TRT apps otherwise we could just compile it as part of |
Beta Was this translation helpful? Give feedback.
-
This is the approach we are targeting All the plugins would be under Case 1: Users only full library. They have a network with pytorch or TensorRT plugins. Since every plugin is built into Case 2: - Users only using runtime library and have a TRT engine. b) Only link Case 3: Standalone TRT network depends on plugins. In this case, users are not using trtorch runtime library. The plugin can be of two different flavours a) nvinfer plugins : they can directly use trtexec or general TRT inference code and load How do we initialize plugin library? option 1 : have a global flag which checks if plugin is initialized in compile time. If so, we don't need to initialize in runtime option 2: Check auto registration of converters. More details will be added on this. |
Beta Was this translation helpful? Give feedback.
-
TRTorch plugin support
Currently we have the following plugins
Interpolate plugin (uses both pytorch aten and TRT layers)
Normalization plugin (in PR, uses pytorch aten calls)
adaptive max pooling (in PR, uses pytorch aten calls)
gelu plugin (in PR, uses TRT plugins)
Instance norm plugin (in PR, uses TRT plugins)
initLibNvInferPlugins
should be used to initialize TensorRT plugins within TRTorch API.Suggested approach
Build a new library called
libtrtorch_plugins.so
. It is the equivalent forlibnvinfer_plugins.so
in TRT.Networks with TRT plugins or plugins with custom cuda kernels
Users should use
libtrtorch_plugins.so
if they are using existing TRT plugins in their network or if they implemented a plugin with their own custom cuda kernel (not using pytorch aten kernels as in plugins 1, 2, 3)Plugins 4 and 5 should be basically compiled into
libtrtorch_plugins.so
. More custom kernels in the future will integrated into this library.Benefit of this:
libtrtorch.so
(is this true ?) andlibtorch.so
during runtime execution. For example, if a user serializes TRT engine (using TRTorch API), for standalone execution of the engine, we need TRT libraries andlibtrtorch_plugins.so
.libtrtorch.so
but network specific widely used ops (for eg: NMS, self-attention layers etc) would fall intolibtrtorch_plugins.so
elu sample converter
app. This can be demonstrated by integrating withlibtrtorch_plugins.so
for users to get started on custom ops.Networks with plugins which use pytorch aten kernels
This usecase requires user to be dependent on torch, trtorch and tensorrt libraries. This would not require linking with
libtrtorch_plugins.so
.Implementation
Currently the entry points for converting a Torchscript module into TRT engine are
trtorch::core::runtime::execute_engine
consumesinputs
and pre-builtserialized_engine
. So during runtime, we also need to initialize nvinfer plugins.TensorRT plugins can be accessed via plugin registry. Sample registration code looks as follows.
We can have a separate component under
core/plugins
which buildslibtrtorch_plugins.so
. By default, when we build TRTorch package we shall buildlibtrtorch.so
andlibtrtorch_plugins.so
. However, we can have a flag that can disable build for plugins if not required (optional).Code under
core/plugins
:register_plugins.cpp
(high level implementation. May differ in actual code.)End-End workflow:
For a network with Gelu plugin (4) or an instance norm plugin (5), TRTorch compilation would be something like this
Question: How do we handle converters for gelu and instance norm ops ? A typical converter op requires you to add
ctx->addPluginV2
. If we have such ops, this may strictly requirelibtrtorch_plugins.so
to be linked withlibtrtorch.so
. A workaround can be implementing converters for plugins also incore/plugins
but this would break the independency oflibtrtorch_plugins.so
withlibtrtorch.so
.Beta Was this translation helpful? Give feedback.
All reactions