Performance Issue with Silero-VAD Model

[Jellun]

Dear Alexander,

I used the example code in silero-vad.ipynb as a guide and applied the silero-vad model to process files sequentially as shown below. I've observed a gradual slowdown in processing speed after each file is completed. To investigate, I queued several identical files. The first file processed returned speech timestamps in about 20 minutes, but as more files were handled, the same file took 2-3 hours to process for timestamps. Could this raise the question of a potential memory leak within the model's library? Are there any methods available to free up resources?

Best regards,
Jun

code in a nutshell:

import torch
torch.set_num_threads(1)

from IPython.display import Audio
from pprint import pprint

SAMPLING_RATE = 16000

USE_ONNX = False

model, utils = torch.hub.load(repo_or_dir='snakers4/silero-vad',
model='silero_vad',
force_reload=True,
onnx=USE_ONNX)

(get_speech_timestamps,
save_audio,
read_audio,
VADIterator,
collect_chunks) = utils

def process_audio(file_queue_manager)
wav = read_audio(file_queue_manager.getfile(), sampling_rate=SAMPLING_RATE)
# get speech timestamps from full audio file
speech_timestamps = get_speech_timestamps(wav, model, sampling_rate=SAMPLING_RATE)
# use speech_timestamps for further process

if name == "main":
# Start consumer thread
consumer_thread = threading.Thread(target=process_audio, args=(file_queue_manager,))
consumer_thread.daemon = True
consumer_thread.start()

[snakers4]

Hi,

We encountered similar problems when running VAD with multiple threads / processes.
#2 (comment)

Please make sure that:

• Each thread has its own instance of the VAD;
• Each thread has PyTorch gradients disabled and 1 PyTorch thread set;

[Jellun]

If I move the model inside the thread like the below:

def process_audio(fileName):
torch.set_num_threads(1)
USE_ONNX = False # change this to True to use onnx model
if USE_ONNX:
#pip install -q onnxruntime
subprocess.run(["pip", "install", "-q", "onnxruntime"])

model, utils = torch.hub.load(repo_or_dir='snakers4/silero-vad',
                                model='silero_vad',
                                force_reload=False,
                                trust_repo=True,
                                onnx=USE_ONNX)

(get_speech_timestamps,
    save_audio,
    read_audio,
    VADIterator,
    collect_chunks) = utils

**# would it be beneficial to model.reset_states() here?**
with torch.no_grad():
    wav = read_audio(fileName, sampling_rate=SAMPLING_RATE)
    # Get speech timestamps from full audio file
    print(f"{datetime.fromtimestamp(time.time()).strftime('%Y-%m-%d %H:%M:%S.%f')[:-3]} - VADing...")
    speech_timestamps = get_speech_timestamps(wav, model, threshold=0.45, sampling_rate=SAMPLING_RATE)
    #...rest of my processing logic

I got the following warning:
C:\Users\JYe\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\nn\modules\module.py:1501: UserWarning: operator () profile_node %669 : int[] = prim::profile_ivalue(%667)
does not have profile information (Triggered internally at ..\third_party\nvfuser\csrc\graph_fuser.cpp:108.)
return forward_call(*args, **kwargs)

More detailed console output:
C:\Users\JYe\source\repos>python WhisperTranscriptionMemory.py
Watch folder: C:\Users\JYe\source\repos\watchFolder

2023-11-04 12:22:51.693 - New audio file detected: C:\Users\JYe\source\repos\watchFolder\16-Mar-2023_01h01m01s01_RLE_NRL_2023_PGM_RD03_MANvPAR_160323_OB_CLEAN.wav
2023-11-04 12:22:52.669 - Process file: C:\Users\JYe\source\repos\watchFolder\16-Mar-2023_01h01m01s01_RLE_NRL_2023_PGM_RD03_MANvPAR_160323_OB_CLEAN.wav

2023-11-04 12:23:37.001 - VADing...
C:\Users\JYe\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\nn\modules\module.py:1501: UserWarning: operator () profile_node %669 : int[] = prim::profile_ivalue(%667)
does not have profile information (Triggered internally at ..\third_party\nvfuser\csrc\graph_fuser.cpp:108.)
return forward_call(*args, **kwargs)

2023-11-04 12:30:31.846 - VAD found: 1897 chunks
2023-11-04 12:30:31.847 - Chunking...
2023-11-04 12:31:35.194 - 89

"""
I believe this warning stems from the Silero-VAD code. Is the warning severe? Could it impact the VAD model's performance? @snakers4, can you provide guidance on resolving this?

[Jellun]

Hi @snakers4 ,

I've been able to suppress the warning by:
import warnings
warnings.filterwarnings("ignore", message=".profile_node.")

However, it would be fantastic for many users if you can confirm that the warning messages above won't cause any performance issue or affect the accuracy of the Silero-VAD model?

Thanks,
Jun