Domain-specific finetuning Whisper

[twardoch]

I am interested in domain-specific fine-tuning.

In my audio, there is a certain number of brand names, person names and domain-specific jargon. Whisper generally transcribes tht text fine but sometimes gets the specific vocab wrong.

Is this the right approach too fix this:

1. Take my audio samples for which I have GT (ground-truth) transcriptions.

2. Run Whisper on them, get the generated text.

3. Compare the generated text with the GT transcripts and gather those that mismatch.

4. Fine-tune Whisper ONLY on the audio samples and the GT transcripts for which a mismatch was found.

So in short, fine-tune only on the corrected errors, not on the entire corpus? I imagine that would reduce the fine-tuning time.

Is there any caveat to my approach?

[glangford]

Before fine tuning it might be useful to experiment using
--initial_prompt 'brand1 brand2 ... term1 term2'
to try and correct for any misspellings you get, and see if accuracy improves to an acceptable level.

[glangford]

• Where do I use --initial-prompt in python code? #355

[JonathanFly]

I haven't tested this, but remember this is also a language model and changing the prompt it also telling Whisper what was said before requested transcription. If the prompt is nothing but a list of keywords, the model is going to transcribe the audio as if it the context of that audio is that it followed list of keywords (instead of being the middle of a conversation, or whatever.) This could impact the quality of the transcription.

So you might want to try a prompt that matches the tone of whatever you are transcribing rather than flat list, but still includes the keywords. "So we were talking about products A,B,C,D..." or something.

Quite possible the impact is so minor this isn't worth thinking about. Worth testing though.

[Lorenzoncina]

This is true. I tested using as initial prompt a list of keywords and it has a bad impact on the WER.

[whicks1]

I’m curious on other folks thoughts here too. Haven’t had great luck in longer files with initial prompt. Was thinking of doing a post process step of running outputs through spaCy or nltk to get named entities and regex to replace.

[twardoch]

Thanks a lot!

[glangford]

Completely anecdotal - in a non-English transcript today (large model, beam search) the surname "von der Leyen" was transcribed as "FonderLion". Running it again with --initial prompt didn't help. Interesting that whisper capitalized it and without a space, so there was at least an attempt at a proper name. :)

[foges]

+1. I agree that having a list of custom vocab would be incredibly useful, likely for most applications. I think use cases for ASR are often domain specific. In my case, I'm using a for dictation and being able to specify names of people I work with and team lingo would be super helpful.