wired inference time for tritonserver

[qiuzhewei]

Description
When I use tritonserver to predict a sequential image list, some wired thing about inference time happend. In most cases, the inference time is about 1ms but some inferece time is way greater(i.e. 29ms). When I change backend to 'onnxruntime', the inference time is stable at around 2ms.

Triton Information
version: 2.29.0
Using the container
To Reproduce
Almost reproducable for every trt model I converted.

Describe the models (framework, inputs, outputs), ideally include the model configuration file (if using an ensemble include the model configuration file for that as well).
Four model in the backend, face detector and landmark model from insighface, two self-trained classification models.

Expected behavior
Stable inference time.

[oandreeva-nv]

Hi @qiuzhewei, thanks for reporting. 2.29.0 seems to correspond to 22.12 version of the container. Could you please verify if this is still an issue for later versions.