How to add new _ALIGNMENT_HEADS?

[JC1DA]

Hi, thanks for your great works.

Just curious how can we add new alignment_heads to adopt some distilled models on huggingface such as distil-large-v2?
( https://huggingface.co/distil-whisper/distil-large-v2 )

The architecture seems to be different so I'm not sure how to use it.

Thanks a lot

[Jeronymous]

That's a very good point! Thx for raising.

Currently there is no user-friendly interface to specify alignment head. I'll think about it.

Do you have an idea how to determine good alignment heads for that distilled models?

[JC1DA]

That's I have no idea :(

[Jeronymous]

I understand from here openai/whisper#1733 (reply in thread) that alignment heads for base models were determined by "manually" inspecting...

For finetuned models, it makes sense to keep the same alignment heads as the base model from which it was adapted.
Not for distilled models...

[Jeronymous]

I added things so that whisper_timestamped can at least work with the distilled model you point

• A trick was missing to load the model in the format required by openai-model
• Concerning the alignment heads, when there is predefined alignment heads, it takes them all from the top 6 layers (at most 6, because in the distilled model from large-v2, there are 2 layers only).
  (Note that things were originally like this, before Jong Wook Kim added alignment heads)

If someone wants to specify special alignment heads, it's possible simply by doing:

import whisper_timestamped

model = whisper_timestamped.load_model(...)
model.alignment_heads = ...
... = whisper_timestamped.transcribe(model, ...)