Replies: 3 comments 10 replies
-
|
Hello - thank you for the question. I am able to reproduce the error in the TorchScript path on the latest Using Latest NightlyCompile Modelimport torch
import torch_tensorrt
...
# Model Definition
...
trt_optimized_module = torch.compile(model,
backend="tensorrt",
dynamic=False,
options={"truncate_long_and_double": True,
"enabled_precisions": {torch.half}})
out = trt_optimized_module(torch.randn((1, 3, 224, 224)).cuda())Regarding TorchScript, it seems that this model also cannot be traced with JIT (only scripted), since |
Beta Was this translation helpful? Give feedback.
4 replies
-
|
Thank you. I installed the following versions: And now I get this error when running These are the available backends: |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the follow-up - could you verify that |
Beta Was this translation helpful? Give feedback.
-
|
Yes it does. import torch_tensorrt
import torchvision
import torch
model = torchvision.models.detection.maskrcnn_resnet50_fpn_v2(weights_backbone=None)
model.cuda()
model.eval()
input_shape = (3, 224, 224)
x = torch.randn(1, *input_shape, device='cuda')
out = model(x)
# Define inputs
inputs = [
torch_tensorrt.Input(
min_shape=[1, *input_shape],
opt_shape=[1, *input_shape],
max_shape=[1, *input_shape],
dtype=x.dtype,
)
]
# Compile to TensorRT
trt_optimized_module = torch.compile(model,
backend="tensorrt",
dynamic=False,
options={"truncate_long_and_double": True,
"enabled_precisions": {torch.half}}) |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the follow-up - based on the version of Torch-TRT you are using ( |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately I have to keep to versions that are available through NGC, so I can't install nightly builds. When I run with |
Beta Was this translation helpful? Give feedback.
5 replies
-
|
Thank you for the follow-up. This error message looks familiar and I believe we addressed a similar issue in a recent PR. Could you share the output of |
Beta Was this translation helpful? Give feedback.
-
|
Sure, the version of |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the reply - I tested out the sample code on the latest Additionally, we are working on the converters for two currently-unsupported operators for this model, |
Beta Was this translation helpful? Give feedback.
-
|
So I'm waiting for either a NGC version that would contain the fix, or for the converter? |
Beta Was this translation helpful? Give feedback.
-
|
The converter is not needed for the model to work; it would just improve performance by reducing segmentation. The NGC container with the fix (or using the |
Beta Was this translation helpful? Give feedback.
-
|
Hello. I'm encountering a similar problem. I'm using the latest Nvidia container (nvcr.io/nvidia/pytorch:24.04-py3), and deploying the same code as above. The compiling part works fine, but the scripting for the TensorScript model fails, with the following error: I'm using this modified code (the first script, before the compiling, works. The second fail): Scripting with example_inputs or with tracing did not work as well. I've also installed MonkeyType in the container. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Hi - thanks for the question. The issue here is the usage of |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Here is a minimal example reproducing the issue:
I get this error:
How do I make this conversion?
Beta Was this translation helpful? Give feedback.
All reactions