Whisper Turbo

[janekpi]

Hi guys!
First of all thanks for such a great tools to work with whispers. I just wanted to ask if are you going to integrate whisper turbo into tensorrt and make it compatible with triton server?

[csukuangfj]

@yuekaizhang

Could you have a look?

I think it should be very easy to support it.

(We have supported it in sherpa-onnx. All we need to do is to convert the whisper turbo to onnx.)

[yuekaizhang]

@janekpi https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/whisper/convert_checkpoint.py#L38 You could follow here. Nothing need to change except model name. We would also update it into sherpa rencently.

[janekpi]

Thank you for reply! Actually we managed to run converted whisper turbo on triton server and even finetuned one. We just encountered an issue with tensorrt_llm version from here https://github.com/NVIDIA/TensorRT-LLM/blob/8681b3a4c0ccc1028bb48d83aacbb690af8f55e7/examples/whisper/requirements.txt and had to downgrade it.

[sh1man999]

Were you able to remove the 30 second limit ?
Maybe there are models in the public domain.
Is onnx faster?