Running on mobile

[Shoshin23]

Hi,

Awesome work with Whisper and many thanks for putting this out there. It definitely will advance a lot of speech-related applications. I'm building a smartglasses tool that helps people with a visual impairment and one of the most preferred forms of interaction for blind users is through voice. I would like to explore Whisper to run on our Android-based Glasses so want to check If there's an Android port in the works? If not, any high-level ideas on how we can adopt this model on run on mobile?

Thanks!

[jongwook]

We haven't tried mobile applications, but you could try TorchScript + PyTorch mobile or ONNX export + ONNX mobile runtime. I expect some difficulties porting the decoding logic, if the mobile environment does not allow running Python.

[szeidner]

I am very interested in a mobile application of this as well - for both iOS and Android. I'm currently running Vosk in a mobile app which works quite well, but does not have the same accuracy as Whisper. @Shoshin23 would you mind sharing if you make any progress in this area? I may give it a try as well.

[edgartaor]

On the other hand, hardware requirements are high. On mobile, I think, you can only use small or medium models. Is that correct?

[Redsandro]

If it helps, python-torch 2.0.1 is on Termux:

https://github.com/termux/termux-packages/tree/master/packages/python-torch

[nyadla-sys]

Build something on top of this to run on Android
https://colab.research.google.com/github/usefulsensors/openai-whisper/blob/main/notebooks/tinynn_pytorch_to_tflite_int8.ipynb

[crazy4pi314]

Oh this is neat! we were looking for ways to get this working on mobile for an OSS app for local transcription for journalists. Do you know if tflite can leverage the special hardware acceleration on apple/google phones?

[nyadla-sys]

Refer below for tflite model files
https://github.com/usefulsensors/openai-whisper/tree/main/models

[nyadla-sys]

@zhuzilin @ArtyomZemlyak
I could generate encoder int8 tflite model from openai->whisper(pytorch) and ran inference.
https://colab.research.google.com/github/usefulsensors/openai-whisper/blob/main/notebooks/tinynn_pytorch_to_tflite_int8.ipynb
can someone help me to generate int8 decoder tflite model from openai->whisper(pytorch)?

[leecrossley]

I got Whisper working on iOS (android is probably easier) by converting the (small) model to CoreML packages in python with the coremltools convert function, as well as writing quite a bit of Swift to them in my scenario. It's not super fast, and the app size is around 500MB using the small model; I haven't looked at the issues on older iOS devices yet (crashing).

A few considerations depend on what you're trying to achieve - there's the speech framework on iOS, for instance (but it recommends an internet connection) which may be faster, more real time, support older devices, reduce app size....

You might want to consider https://github.com/snipsco/snips-platform-android-demo as an option reading your case - just throwing it out there for evaluation

Here's a fun little post about my adventure so far but happy to have a chat to see if I can help: https://sandigital.uk/blog/openai-whisper-scottish-accent/

[jrappeneker]

Hi, would you be willing to share your code? I'm working on an app to perform linguistics research and it would be really helpful!

[Alirsgp]

I'd also love a way to see this work. Have some use cases in mind.

[leecrossley]

I'll get this shared as soon as I get a chance, hopefully this week

[Alirsgp]

Thanks lee. I'm a bit stuck on the CoreML conversion process.

[Shabbir7999]

@leecrossley Can you please share your code for Android or IOS with me?

[nyadla-sys]

encoder/decoder tflite(int8) models can be generated with the below script
https://colab.research.google.com/github/usefulsensors/openai-whisper/blob/main/notebooks/tinynn_pytorch_to_tflite_int8.ipynb

[nyadla-sys]

here is the colab link to generate whisper.tflite file
https://colab.research.google.com/github/usefulsensors/openai-whisper/blob/main/notebooks/tflite_from_huggingface_whisper.ipynb

Updated input shapes using input signature to create a new hybrid whisper.tflite 40MB model (activations are in float32, weights are in int8). I was then able to execute inference with dummy input on whisper.tflite using minimal example from TFLite C++ API on a Linux PC.
Follow the below instructions to build minimal binary on Linux
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/examples/minimal

run the below command to run inference on whisper.tflite
./minimal ~/Desktop/whisper.tflite

Next steps:Feed the audio input->generate spectrograms->feed to model->post process model output in c++ code

[nyadla-sys]

Please follow the README to run whisper.tflite using tflite framework
https://github.com/usefulsensors/openai-whisper

[Topping1]

Hi,

Awesome work with Whisper and many thanks for putting this out there. It definitely will advance a lot of speech-related applications. I'm building a smartglasses tool that helps people with a visual impairment and one of the most preferred forms of interaction for blind users is through voice. I would like to explore Whisper to run on our Android-based Glasses so want to check If there's an Android port in the works? If not, any high-level ideas on how we can adopt this model on run on mobile?

Thanks!

In this repository you can find a Flutter port of Whisper: https://github.com/azkadev/whisper_dart
it is based on the C++ port of whisper: https://github.com/ggerganov/whisper.cpp

[nyadla-sys]

I developed Android APP based on whisper.tflite (quantized ~40MB) running inference in ~2 seconds for 30 seconds audio clip
https://github.com/usefulsensors/openai-whisper/blob/main/android_app/release/app-release.apk
Use the below command to push on to android phone
adb install -r -t ~/android_app/release/app-release.apk

[View-my-Git-Lab-krafi]

I developed Android APP based on whisper.tflite (quantized ~40MB) running inference in ~2 seconds for 30 seconds audio clip https://github.com/usefulsensors/openai-whisper/blob/main/android_app/release/app-release.apk Use the below command to push on to android phone adb install -r -t ~/android_app/release/app-release.apk

Nice but looks like it's only for English

[nyadla-sys]

I released Whisper Android App based on Whisper.tflite ~40MB quantized model to the Android App store.

Link to Android app store
https://play.google.com/store/apps/details?id=com.whisper.android.tflitecpp

[nyadla-sys]

This is only a proof-of-concept project to create an Android app based on Whisper TFLite, which leverages the stock Android UI to show off its features.
Whisper-TFLIte-Android-Example

[nyadla-sys]

https://apps.apple.com/in/app/whisper-asr/id6444556326

Pls download iOS app from apple app store which uses whisper tflite model

[amgaa]

Hey, awesome work there!

I was just wondering if you plan to extend your iOS app to other languages. It would be great to have the app available in more languages for a wider audience. Also, have you considered opening up the source code for the app for others to build their own models? Just curious about your thoughts on this. Thanks!

[cvladan]

Yes, would it be possible for us to test a multilingual model to see how it performs? Thank you for considering this request.

[cvladan]

On Android

[nyadla-sys]

Please feel free to download the openai/whisper-tiny tflite-based Android APP from Google App Store.
https://play.google.com/store/apps/details?id=com.whisper.android.tflitecpp

[nyadla-sys]

Please refer the below link which uses tflite as part of AndroidVoice keyboard
https://github.com/MichaelMcCulloch/WhisperVoiceKeyboard

[MichaelMcCulloch]

@nyadla-sys I am attempting to move to the large model, and run inference on NNAPI, and I am very much stuck. I don't suppose you would be able to lend your brain and eyes?

[nyadla-sys]

@MichaelMcCulloch are you able to generate large whisper tflite model?if so what kind of issues you are seeing.
Please give me more details along with adb logcat I can help on this

[MichaelMcCulloch]

Let's move this conversation to an issue on my repo.

[slurpey]

I would be definitely willing to pay for your app if it had an automated copy of the text to the clipboard or a big copy button at the bottom, so I could use it as a replacement for Google Voice Typing.

[SubhamSubhasisPatra]

Hey @nyadla-sys, I just came across your application it's awesome, and I want to integrate it into a react native application. Can u plz give me some clues on how to integrate it?

[nyadla-sys]

You may find the source code for a Linux C++ application and an Android Java app on my Github repository at the following link: https://github.com/usefulsensors/openai-whisper.

[SubhamSubhasisPatra]

@nyadla-sys, This .tflite will be easy to use directly in an Android application, but is there any way to create the models.json and weight.bin file from that, because react native does not support .tflite?

[zhanghui19880625]

hello, Why does the whisper_android app run above android 9.0?
I want to one can run above android 7.0

[nyadla-sys]

Android example to use whisper tflite model https://github.com/nyadla-sys/whisper.tflite/tree/main

[tawanda-profuse]

Which app are you referring to specifically? Please share the link.

…

On Sat, 03 Feb, 2024, 11:10 slurpey, ***@***.***> wrote: I would be definitely willing to pay for your app if it had an automated copy of the text to the clipboard or a big copy button at the bottom, so I could use it as a replacement for Google Voice Typing. — Reply to this email directly, view it on GitHub <#11 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGBLNUDAYXZY2TTD2JCNAU3YRX5IZAVCNFSM6AAAAAAQSNDAMCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DGNJTGYZDA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[slurpey]

It's the one above my reply trail: https://play.google.com/store/apps/details?id=com.whisper.android.tflitecpp

[adamlogan]

Link is dead.

[View-my-Git-Lab-krafi]

i dont know do anyown want to use whisper using web browser . and your pc as server https://gitlab.com/krafi/whisperweb