Add triton server #3
Conversation
|
Thanks! I am looking at it. |
| ) | ||
| from torch import Tensor, nn | ||
|
|
||
| #from icefall.utils import make_pad_mask |
There was a problem hiding this comment.
This conformer_triton.py is mainly for remove the k2 dependency. Currently, only support greedy_search_batch, so I didn't install k2. It could be removed after dockerfile improvement with k2, lhoste install.
| bias=False, | ||
| ) | ||
|
|
||
| def forward(self, y: torch.Tensor) -> torch.Tensor: |
There was a problem hiding this comment.
remove need_pad parameters, since it always false when infer, so we don't need to prepare input_need_pad for triton.
| self.decoder_proj = ScaledLinear(decoder_dim, joiner_dim) | ||
| self.output_linear = ScaledLinear(joiner_dim, vocab_size) | ||
|
|
||
| def forward( |
There was a problem hiding this comment.
remove projected_input here, I didn't have ForwardEncProj and DecProj outside joiner. Otherwise, it would add two more submodules under triton and make things complicated.
|
|
||
| return logit | ||
|
|
||
| class AttributeDict(dict): |
There was a problem hiding this comment.
These duplicate functions below also for avoid k2 dependency. When we add more decoding algorithms like fast_beam_search we would not need these duplicate functions.
| self.vocab_size = sp.get_piece_size() | ||
| self.sp = sp | ||
|
|
||
| def execute(self, requests): |
There was a problem hiding this comment.
This is the start of greedy_beam_sarch_batch.
| value = value["string_value"] | ||
| if key == "num_mel_bins": | ||
| opts.mel_opts.num_bins = int(value) | ||
| # elif key == "frame_shift_in_ms": |
There was a problem hiding this comment.
I would suggest either removing frame_shift and frame_length from the config file, or reading them if the config file provides them.
There was a problem hiding this comment.
Sure, I would uncomment them.
| @@ -0,0 +1,144 @@ | |||
| import triton_python_backend_utils as pb_utils | |||
There was a problem hiding this comment.
Do you need to install https://github.com/triton-inference-server/python_backend?
Its readme.md says you have to install it in the docker image.
Correct me if I am wrong.
There was a problem hiding this comment.
I think we don't need to install it. Its readme looks like install a python_backend example. Just tritonserver official docker could offer all triton ensentials we need.
| speech[i, 0: f_l, :] = f.to(self.output0_dtype) | ||
| speech_lengths[i][0] = f_l | ||
| # put speech feature on device will cause empty output | ||
| # we will follow this issue and now temporarily put it on cpu |
There was a problem hiding this comment.
Sorry, it was triton issue. I have deleted them and tested.
| }, | ||
| { | ||
| key: "bpe_model", | ||
| value: { string_value: "/ws/yuekaiz/icefall-asr-librispeech-pruned-transducer-stateless3-2022-04-29/data/lang_bpe_500/bpe.model"} |
There was a problem hiding this comment.
Is this path available in the docker container?
There was a problem hiding this comment.
Sorry, would update it.
| { | ||
| name: "encoder_out__0" | ||
| data_type: TYPE_FP32 | ||
| dims: [-1, 512] # [-1, feature_size] |
There was a problem hiding this comment.
| dims: [-1, 512] # [-1, feature_size] | |
| dims: [-1, 512] # [-1, encoder_out_dim] |
triton/client/utils.py
Outdated
| @@ -0,0 +1,59 @@ | |||
| import numpy as np | |||
There was a problem hiding this comment.
This file has no copyright info and it is not clear where it is from.
Could you use https://github.com/pzelasko/kaldialign
to calculate the WER instead of reinventing the wheel.
We are alreadying using it in icefall.
| @@ -0,0 +1,127 @@ | |||
| # Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved. | |||
There was a problem hiding this comment.
Comment for the file triton/client/1089-134686-0001.json
Where is it used? Can we remove it?
There was a problem hiding this comment.
It's generated by "python3 generate_perf_input.py test_wavs/1089-134686-0001.wav <input.json>", which is used by perf_anyalzer. I would remove it and update the reademe.md.
|
@csukuangfj Thanks for your review. I have commited changes according to your comments, would you mind checking them? |
|
Thanks! Looks good to me. Is it ready to merge? |
I think so. I would like add benchmark results in next PR with features like fast beam search algorithms. So we could also have a comparison of WER, throughput. Also, I would update dockerfile to support k2 for fastbeam search. |
I added this triton server and client for offline ASR. Moslty following wenet triton server https://github.com/wenet-e2e/wenet/tree/main/runtime/server/x86_gpu
TODO (Maybe after this PR?)