Just out of curiosity, does anyone know how much GPU resources and time OpenAI used to train Whisper?

[maowandong]

Just out of curiosity, does anyone know how much GPU resources and time OpenAI used to train Whisper?

I am fine-tuning whisper-large-v2 for chinese with 10,000+ hours corpus (15 million audio-label pair) on A100 x 8, i tooks 8-9 secons per iteration, and processing a total of 3 epochs takes 18-20 days.

[glangford]

I'm not aware of publicly available information on the GPU resources used, but the Whisper paper describes a corpus of 680,000 hours with training details for the different models given in section 2.4.

https://arxiv.org/abs/2212.04356

[mzamini92]

[maowandong]

thanks, Fine-tune Whisper-Large-v2 for Chinese with 10000+ hours(15 million audio-label pairs) seems cost more than two weeks for me on 8XA100~

[phineas-pta]

The Whisper-Large model has 137B parameters

u mistook something else entirely, the large model has <1.6B params

A single A100 GPU has 40GB

OP never said it's 40gb or 80gb, that's a hell of difference

Assuming that OpenAI used a cluster of 100 A100 GPUs

where do u get this number ?

also it depends on batch size, very likely that openai has a lot more of a100 so they could use bigger batch size than OP