Voice Activity Detection

[vincentqb]

I'd love to hear from our users what is the impact of their favorite VAD algorithm on their model's performance at both training and inference (offline, online, streaming).

• Is there one we should add in torchaudio?
• Does anyone uses webrtc's vad implementation mentioned in comment?

We currently have in torchaudio

• the example interactive ASR pipeline
• sox re-implementation in native pytorch
• apply_effects_tensor with VAD effect

cc @mthrok @astaff

[PetrochukM]

Voice Activity Detection has been important for my work TTS and STT. It's useful for segmenting large audio files before training. I'd love a basic VAD implementation akin to:
https://maelfabien.github.io/project/Speech_proj/#pros-and-cons

webrtc didn't work for me because it didn't expose enough parameters for tunning. The default version of webrtc didn't work well for clean datasets... It tended to overcorrect for noise. It'd cut off unvoiced consonants because it thought they were noise... I tried all of the available settings in py-webrtcvad to correct for this.

I don't think there is one perfect algorithm for VAD because you need to make all sorts of assumptions about the SNR ratio. So, I prefer a VAD which can be tuned by hand for different situations: no noise, minimal background noise, medium background noise, etc... and speaker levels: whisper, conversational, speech, etc...

The one currently in torchaudio didn't work for me because it is focused on audio trimming instead of detecting voice throughout the audio.

[vincentqb]

great response @PetrochukM and thanks a lot for the input :) have you also tried this one?

[vincentqb]

(quick note: some elements of the algorithm you suggested are also similar to one of the pitch detection algorithm we have)

[PetrochukM]

Oh. I didn't. Thanks for the tip! It looks pretty close to what I was describing :)

Also, another issue that we had with VAD was memory. For some reason, scipy doesn't support memmap. This made it particularly difficult to work with files longer than 1 hour.

I actually tried to use the pitch detection algorithm in torchaudio but it was just too slow compared to other algorithms. I don't have too many details right now but it was much slower than something like loudness detection. I can give more details, in another thread, sometime later!

[vincentqb]

@PetrochukM, have you tried the kaldi voice activity detection? also mentioned in pytorch forum

[PetrochukM]

I considered it but I decided against it during my research. I vaguely remember being overwhelmed by the complexity of it. I hope my POV helps!

[vincentqb]

I hope my POV helps!

Definitely, thank you :)