❓ [Question] TRTorch and Pytorch Serve

[p1x31]

❓ Question

Is it possbile to use TRTorch with TorchServe?
If not what is the best way to deploy TRTorch programs?

What you have already tried

In the documentation, it is said I can continue to use programs via PyTorch API
I have converted all my models to TRTorch.

Environment

Build information about the TRTorch compiler can be found by turning on debug messages

• PyTorch Version (e.g., 1.0):
• CPU Architecture:
• OS (e.g., Linux):
• How you installed PyTorch (conda, pip, libtorch, source):
• Build command you used (if compiling from source):
• Are you using local sources or building from archives:
• Python version:
• CUDA version:
• GPU models and configuration:
• Any other relevant information:

Additional context

[narendasan]

I think you should be able to with some slight modifications to torchserve. Basically you just need to load the TRTorch runtime in your handler and also insure that the input size to your model is in the range specified at compile time. I tried out creating a mar file from a TRTorch compile TorchScript file from the torch-model-archiver tool and that seemed to work. Let us know if you try it and if you are successful or hit any issues, we'd love to hear about it

[p1x31]

Thanks for the heads up. I'll close this question for now and will let you know the results.

[mrcolo]

Hi! I'm experiencing a weird problem with Torch_TENSORRT - it seems that within a local script environment, performing inference with my model and its correspective RT version consists of a 3x speed up in inference time, which is awesome.

The problem is that I can't replicate this speed up within torchserve. Even though I load the runtime in my handler (import torch_tensorrt), inference with torchserve appears to take more time than the torchscript version of the same model.

Is someone experiencing the same problem?

[fancoltran]

have you resolve it?

[narendasan]

Can you share instructions on how to replicate your setup? @mrcolo

[fancoltran]

i'm sorry i'm not Meta dev. I'm also get struggled into this and ask you for solving problem