Much slower via command line tool than in Python?

[ThaddeusB]

Hello,

I noticed the command line tool is roughly half the speed of running Python code in the following comparison:

Command line:
whisper file.mp3 --model large

Python:
import whisper

model = whisper.load_model("large")
result = model.transcribe("file.mp3")
print(result["text"])

If the command line is just slower, than that's good to know. If the two commands are ways are actually doing something different, can someone explain what the difference is?

Thanks!

[FurkanGozukara]

Did they generate same output? Python perhaps using lower size model? My CPU and GPU gets maximized via CMD. I don't think there can be such speed difference.

[jongwook]

The command line has different default settings on some hyperparameters, where the command line tool defaults to beam search and model.transcribe() will do greedy decoding by default. It'll become equivalent if you do:

model.transcribe("file.mp3", beam_size=5, best_of=5)

greedy decoding is faster but is more likely to get stuck in repetition loops, and in general slightly less accurate.

[jongwook]

The default settings are what we empirically found to be working consistently across different types of recordings, but manual tuning may be required in case it fails with repetition looping or hallucination.

[shervinemami]

Just for future reference, I want to mention that only one of "beam_size" or "best_of" can actually be used by the engine. When you use transcribe(f, beam_size=5, best_of=5) it will silently perform transcribe(f, beam_size=5, best_of=None), whereas if you use decode(f, beam_size=5, best_of=5) directly then it will give an exception because you can't actually use beam_size and best_of at the same time.

[FurkanGozukara]

@shervinemami are there any documentation that gives detailed explanation about each hyper parameter such as best_of

[shervinemami]

Hi @FurkanGozukara , best_of is briefly described in the command line interface, meaning you can access it from the terminal, without writing code:

whisper/whisper/transcribe.py

Line 264 in 9f70a35

parser.add_argument("--best_of", type=optional_int, default=5, help="number of candidates when sampling with non-zero temperature")

Whereas if you're using whisper from Python code, you would use the DecodeOptions parameters, briefly described here:

whisper/whisper/decoding.py

Line 79 in 9f70a35

best_of: Optional[int] = None # number of independent samples to collect, when t > 0

[FurkanGozukara]

@shervinemami ty but still doesnt give me detailed idea. e.g. : number of beams in beam search, only applicable when temperature is zero. what is beam size and beam search? what effect does it have?

[jcdoster]

So, my command line executions are just slower. Keeping Temperature at 0.0, Beam_size at 1, for a 17 second .wav clip, I have an execution time of 18 seconds on the command line and 8 seconds within the Spyder PYTHON IDE. I've tried every other option parameter, that's what I get, a difference time factor of 2, the command line being slower. I don't understand, should be reversed.

[jwnacnud]

Does it get progressively slower than more look up see you do or is it just slow every time even after you haven’t been using it for a while?

On Fri, Mar 10, 2023 at 2:32 PM Jason Doster ***@***.***> wrote: So, my command line executions are just slower. Keeping Temperature at 0.0, Beam_size at 1, for a 17 second .wav clip, I have an execution time of 18 seconds on the command line and 8 seconds within the Spyder PYTHON IDE. I've tried every other option parameter, that's what I get, a difference time factor of 2, the command line being slower. I don't understand, should be reversed. — Reply to this email directly, view it on GitHub <#177 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGW5AZ5LCU2KVNJG7TD4SLW3OMV5ANCNFSM6AAAAAAQYLB6FI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Jeffrey Duncan