Add audio to text pipeline

[eliteprox]

This change adds the audio-to-text pipeline to the AI Runner

[ad-astra-video]

I did a review of this PR. Initial comments below.

My tests included: speech-mp3-lowbitrate.mp3 (worked), speech-aac-lowbitrate.m4a (worked), vp8-opus.webm (worked), vp8-vorbis.webm (worked), vp9-vorbis.webm (worked), h264 and h265 variants did not work for some reason in mp4 files. Pulling the audio out of the mp4 files were successfully processed.

• Communicate processing was not successful when input is in wrong format or failure happens

  • I was getting an error on some transcoded files that were intentionally in different input format. The results right now is a null/empty text response: {"chunks":null,"text":""}. I think should be an error response code or at least an error in the response to provide error or "ok" if no error. The last line in the traceback below is where transformers is trying to use ffmpeg in container to parse the file I believe.

  • I checked and found that FFmpeg is installed in the runner container but is an ancient version 4.2.7.

    File "/root/.pyenv/versions/3.11.9/lib/python3.11/site-packages/transformers/pipelines/pt_utils.py", line 186, in __next__ processed = next(self.subiterator) ^^^^^^^^^^^^^^^^^^^^^^ File "/root/.pyenv/versions/3.11.9/lib/python3.11/site-packages/transformers/pipelines/automatic_speech_recognition.py", line 362, in preprocess inputs = ffmpeg_read(inputs, self.feature_extractor.sampling_rate) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.pyenv/versions/3.11.9/lib/python3.11/site-packages/transformers/pipelines/audio_utils.py", line 41, in ffmpeg_read raise ValueError( ValueError: Soundfile is either not in the correct format or is malformed. Ensure that the soundfile has a valid audio file extension (e.g. wav, flac or mp3) and is not corrupted. If reading from a remote URL, ensure that the URL is the full address to **download** the audio file.

• Is seed a parameter for the speech-to-text pipeline? I was not seeing it on Whisper, maybe keeping it in case a model would use it?
• Add the download cmd to dl_checkpoints.sh
  • huggingface-cli download openai/whisper-large-v3 --include "*.safetensors" "*.json" --cache-dir models

[eliteprox]

I've updated the runner to handle these errors.

The ai-runner responds with a 400 Bad Request and logs 2024-07-02 15:52:18,181 INFO: 172.17.0.1:38806 - "POST /speech-to-text HTTP/1.1" 400 Bad Request

The gateway logs I0702 11:50:40.632077 2527064 ai_process.go:561] clientIP=192.168.10.71 request_id=dd12552c Error submitting request cap=31 modelID=openai/whisper-large-v3 try=6 orch=https://0.0.0.0:8936 err=speech-to-text container returned 400

The orchestrator logs 2024/07/02 11:50:40 ERROR speech-to-text container returned 400 err="{\"detail\":{\"msg\":\"Error processing audio file: Soundfile is either not in the correct format or is malformed. Ensure that the soundfile has a valid audio file extension (e.g. wav, flac or mp3) and is not corrupted. If reading from a remote URL, ensure that the URL is the full address to **download** the audio file.\"}}"

Attached is 400 Bad Request response from Swagger UI:

[eliteprox]

• I was getting an error on some transcoded files that were intentionally in different input format. The results right now is a null/empty text response: {"chunks":null,"text":""}. I think should be an error response code or at least an error in the response to provide error or "ok" if no error. The last line in the traceback below is where transformers is trying to use ffmpeg in container to parse the file I believe.

I've added error handling to the AI Runner if the model experiences any error while processing the file. It specifically checks for
"invalid soundfile" and returns these as 400 Bad Request.

• Is seed a parameter for the speech-to-text pipeline? I was not seeing it on Whisper, maybe keeping it in case a model would use it?

That's correct, I don't see a purpose for the seed parameter with this model. I think it's unlikely this pipeline will need it.

• Add the download cmd to dl_checkpoints.sh

  • huggingface-cli download openai/whisper-large-v3 --include "*.safetensors" "*.json" --cache-dir models

I've added this to dl_checkpoints.sh