❓ Calling Silero VAD model from Huggingface datasets.map

[wjassim]

Hello,

I’m attempting to utilize the Silero VAD model from a huggingface-based dataset using the map function and multiple workers for parallel processing. Here’s the relevant code snippet:

import torch
from datasets import Dataset, load_dataset, Audio
torch.set_num_threads(1)

def process_row(self, row, index, model, get_speech_timestamps, collect_chunks):
    # read audio file (wav) saved in row["ref_wave"],
    # apply VAD
     speech_timestamps = get_speech_timestamps(wav, model, sampling_rate=16000)
    return {'speech_timestamps ': speech_timestamps}

model, utils = torch.hub.load(repo_or_dir='snakers4/silero-vad',
                              model='silero_vad',
                              force_reload=True,
                              onnx=USE_ONNX)

(get_speech_timestamps, _, _, _, collect_chunks) = utils

# load data
dataset = load_dataset("csv", data_files='path_to_csv', split="train")

dataset = dataset.map(
            self.process_row,
            num_proc=8,
            with_indices=True,
            batched=False,
            remove_columns=dataset.column_names,
            fn_kwargs={
                "model_vad": model,
                "get_speech_timestamps": get_speech_timestamps,
                "collect_chunks": collect_chunks,
            },
        )

However, I’m encountering an issue that generates the following error message:

...transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, 
 a random hash was used instead. Make sure your transforms and parameters are serializable with pickle 
 or dill for the dataset fingerprinting and caching to work. 
.....
RuntimeError: Tried to serialize object __torch__.vad.model.vad_annotator.VADRNNJITMerge which 
does not have a __getstate__ method defined!

I'm unsure about how to correctly pass the VAD model to the 'datasets.map' function and make it run with multiprocessing. Any assistance you can provide would be greatly appreciated. Thank you for your time. Best regards.

[snakers4]

and make it run with multiprocessing

The correct low-level way to run VAD with multiple python processes is as follows:

• Have a separately run init function, that is invoked at the start of EACH process. This function should load the VAD and set the number of CPU threads for PyTorch if necessary. Same for ONNX. Please note that both PyTorch and ONNX models are not python objects, but merely pointers to the underlying objects;

• Python has may APIs for multiprocessing (Process, ProcessPool, ProcessPoolExecutor, etc etc). Many of them have a custom init function parameters. The key here is NOT to reuse the same pointer, but to create truly separate model instances for each process;

I’m attempting to utilize the Silero VAD model from a huggingface-based dataset using the map function

I am not sure how it works, but looks like it pickles the objects it passes to sub-processes. You have 4 options:

• Looks for similar custom init option and / or rewrite their class;
• Use some sort of queue / messaging system / sub process;
• Write the parallelization bit from scratch, try different spawning strategy;
• Pre-process data and store annotation in advance;

[wjassim]

Thank you very much for the detailed feedback.

Yeah I noticed an example in your repository for initializing a function at the beginning of each process. I also saw your discussion regarding a Pull Request.

I shared the same issue on the Hugging Face community site, and one of the developers kindly suggested serializing the model just before running the map function using the following code:

import copyreg
import os

def pickle_model(model):
  if not os.path.exists("model_scripted.pt"):
    model.save("model_scripted.pt")
  return torch.jit.load, ("model_scripted.pt",)

copyreg.pickle(type(model), pickle_model)

I followed this method, but I encountered another issue with the error message:

 ....\lib\site-packages\dill\_dill.py", line 432, in find_class return StockUnpickler.find_class(self, module, name)
ModuleNotFoundError: No module named 'utils_vad' 

I am not sure why this error is occurring. I will check the options you kindly suggested and will provide updates once I've made progress.

Thank you again,
Best regards.

[snakers4]

ModuleNotFoundError: No module named 'utils_vad'

The objects pulled are not only a PyTorch saved model (a pickle with dicts), but a set of functions + a JIT model.

model.save("model_scripted.pt")

This most likely implies a PyTorch checkpoint / saved model, but we have JIT models and / or ONNX models.

[wjassim]

I got it. Thank you so much for your feedback. Best regards.