Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use silero v3.1 #142

Merged
merged 6 commits into from
Nov 30, 2023
Merged

Use silero v3.1 #142

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
f961580
Use silero v3.1
Jeronymous Nov 18, 2023
52100d2
Fixes #139 : Load distilled models, and assign them default alignment…
Jeronymous Nov 26, 2023
7bead93
version 1.14.0 : can choose among several VAD methods (different vers…
Jeronymous Nov 30, 2023
a9334c2
update unittests
Jeronymous Nov 30, 2023
c7a71e4
Merge branch 'master' into bugfix/use_silero_v3.1
Nov 30, 2023
689e07e
version 1.14.1 : more hacky but solid way of handling silero folder m…
Jeronymous Nov 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions whisper_timestamped/transcribe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
import gzip, base64
import copy
import re
import shutil

# Constant variables
from whisper.utils import format_timestamp
Expand Down Expand Up @@ -1794,12 +1795,27 @@ def get_vad_segments(audio,
if silero_vad_model is None:
import onnxruntime
onnxruntime.set_default_logger_severity(3) # Remove warning "Removing initializer 'XXX'. It is not used by any node and should be removed from the model."
repo_or_dir = os.path.expanduser("~/.cache/torch/hub/snakers4_silero-vad_master")
repo_or_dir_master = os.path.expanduser("~/.cache/torch/hub/snakers4_silero-vad_master")
repo_or_dir_v31 = os.path.expanduser("~/.cache/torch/hub/snakers4_silero-vad_v3.1")
repo_or_dir = repo_or_dir_v31
source = "local"
tmp_folder = None
if os.path.exists(repo_or_dir_master):
tmp_folder = repo_or_dir_master + ".tmp"
shutil.move(repo_or_dir_master, tmp_folder)
# Make a symlink to the v3.1 model, otherwise it fails
os.symlink(repo_or_dir_v31, repo_or_dir_master)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I don't make a symbolic link snakers4_silero-vad_master -> snakers4_silero-vad_v3.1 in ~/.cache/torch/hub/, it does not work.
It fails with

  File "/home/jlouradour/.cache/torch/hub/snakers4_silero-vad_v3.1/utils_vad.py", line 16, in __init__
    self.session = onnxruntime.InferenceSession(path)
  File "/home/jlouradour/.local/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 360, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "/home/jlouradour/.local/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 397, in _create_inference_session
    sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.NoSuchFile: [ONNXRuntimeError] : 3 : NO_SUCHFILE : Load model from /home/jlouradour/.cache/torch/hub/snakers4_silero-vad_master/files/silero_vad.onnx failed:Load model /home/jlouradour/.cache/torch/hub/snakers4_silero-vad_master/files/silero_vad.onnx failed. File doesn't exist

I also don't want to rename "v3.1" into "master" so that both versions can co-exist.
So I temporarily move master to make the symbolic link so that it works.
But I have to it at every call of the VAD. This is awkward and... imagine user does not have write permission on the folder...

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Jeronymous
I found a commit that addresses this issue in silero repository. But judging from commit dates, it seems to have been merged after default switched to v4.0? I don't know what is the best option here. I personally don't use the silero repo anymore. Because I wanted a near-instant inference start on demand (to skip non-speech in my local mpv player from any playback position) I switched to a self-contained minimal C program that calls to onnxruntime's C api in a dll. I just pipe the audio from ffmpeg and it immediately returns the timestamps. Switching silero versions for me was just a matter of renaming the model file and adjusting the onnxruntime api (the V4 IIRC changed output tensor dimension).

snakers4/silero-vad@df1d520

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for sharing this.

Given this, my solution in python makes sense. Just I don't like the fact of meshing like this with folders when processing an audio.
Maybe the right solution is just to fork silero from v3.1 and fix that particular issue inside...

Concerning the quality of silero VAD, apparently, there is no free lunch : it's not obvious that solving this case would not degrade in other circumstances (speech with very noisy background). See comments here: snakers4/silero-vad#396 (comment)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Jeronymous In my case, the hypothetical issues of detecting speech with very noisy background accurately are so rare that they are indeed hypothetical for me. I can't speak for others using the whisper-timestamped, and I don't know your intended main usage for the project, so I will offer a suggestion strictly from my own perspective and usage patterns. For my use cases I would settle on the silero-vad version that suits best for my needs and set that as a reasonable default, and implement it first and foremost. Support the other version later if there is enough demand. And since silero-vad models are so small, and the inference code is so simple and short, I would just bundle the models (if license permits) and inference code with the whisper-timestamped project, and by removing a dependency we get rid of a potential failure point, downloading the entire repository for like 50 lines of python code, and dealing with all the symlinking and paths nonsense.
Unfortunately it wouldn't address my main issue with the project, (unacceptably high VRAM usage, making it impossible to run whisper-v3-large), but that's better addressed in topical issues like #110

if not os.path.exists(repo_or_dir):
repo_or_dir = "snakers4/silero-vad"
# Load version 3.1 from 17/12/2021 -- see https://github.com/snakers4/silero-vad/wiki/Version-history-and-Available-Models
# because of problems with version 4, see https://github.com/linto-ai/whisper-timestamped/issues/74
repo_or_dir = "snakers4/silero-vad:v3.1"
source = "github"
silero_vad_model, utils = torch.hub.load(repo_or_dir=repo_or_dir, model="silero_vad", onnx=True, source=source)
os.remove(repo_or_dir_master)
if tmp_folder:
shutil.move(tmp_folder, repo_or_dir_master)
assert os.path.isdir(repo_or_dir_v31)

silero_get_speech_ts = utils[0]

# Cheap normalization of the volume
Expand Down