AI Audio Transcriber

A minimalistic application to generate transcriptions for audio built using Python

🚀 Demo

v.0.0.1

v.0.0.2 (Transcribing a Youtube Video Explaining Whisper)

v.0.0.2 (Transcribing an English Song - Thinkin About It)

v.0.0.3 (Transcribing a clip from Lex Fridman's podcast)

v.0.0.4 (Transcribing another clip from Lex Fridman's podcast)

📝 Basic Application WorkFlow

flowchart LR 
    U([Cliemt])
    
    I{Choose\n Input Mode}
    U -----> I
    
    I1[YouTube Video URL] 
    I2[Upload Video File]
    I3[Upload Audio File]
    I ---> I1 & I2 & I3

    YTC{"Check if\n Audio is available?"}
    YTA("Download video\n from YouTube")
    YTV("Download video\n from YouTube")
    
    I1 ---> YTC
    YTC --yes---> YTA
    YTC --no---> YTV

    VTA["Convert Video to Audio"]
    YTV ---> VTA
    I2 ---> VTA

    LA["Load Audio File"]
    YTA & VTA & I3---> LA
    
    M{"Choose\n Model Type"}
    U -----> M

    M1[(Ramanujan)]
    M2[(Bose)]
    M3[(Raman)]
    M4[(Kalam)]
    M ---> M1 & M2 & M3 & M4

    LM[Load Relevant Whisper Model]
    M1 & M2 & M3 & M4 --> LM

    GT("Generate Transcripts")
    LA & LM ---> GT

    O1(["Detected \n Language"])
    O2(["Complete \nSubtitle Text"])
    O3(["Subtitles \nwith Timestamps"])
    GT ---> O1 & O2 & O3

    OF(["Original\n Audio or Video"])
    D{{"Display to Client"}}
    I ---> OF
    O1 & O2 & OF ---> D

    DO{"Choose\n Output Option"}
    D1["SRT\n File"]
    D2["VTT\n File"]
    D3["Text\n File"]
    DP["Process Subtitle Object"]
    DN{{"Download Button"}}

    O3 ---> DP
    U ---> DO
    DO ---> D1 & D2 & D3 ---> DP ---> DN

    subgraph Result
        D
        DN
    end

Loading

🥊CI/CD

(Preferred Pipeline Using GitHub Actions for Docker Image)

⚒️ Set-Up Instructions

Open your terminal / command prompt.

Clone the repository

git clone https://github.com/smaranjitghose/AIAudioTranscriber.git

Change the directory to the cloned project
```
cd AIAudioTranscriber
```

A. Without using Docker

Ensure you have any version of Python below 3.10 installed in your system and you have virtualenv package installed
```
which python
```
```
pip install virtualenv
```
Create a new virtual environment
```
python -m venv env
```

Activate virtual enviroment

On Mac/Linux
```
source env/bin/activate
```
On Windows
```
env/Scripts/Activate.ps1 
```

Install ffmpeg in your local syste,
- On Windows using Chocolatey
```
choco install ffmpeg
```
- On MacOS using Homebrew
```
brew install ffmpeg 
```
- On Debian/Ubuntu
```
sudo apt update && sudo install ffmpeg
```
- On Arch Linux
```
sudo pacman -S ffmpeg 
```
Install the dependencies
```
pip install -r requirements.txt
```
Download the model weights (This will take a few minutes since the total size of models in gigabytes)
```
python get_model_weights.py
```
Run the Web application
```
streamlit run .\Home.py
```
Note:
- If the app does not load by itself in your default browser, open a browser of your choice and navigate to http://localhost:8501
- To stop the application, press CTRL + C in your terminal

B. Using Docker

Make sure you have Docker installed on your system. Refer the documentation here if you need assistance setting up.
Build a docker image
```
docker run -t aitranscriber:v0.0.4 .
```
Note:
- You may give any name instead of aitranscriber and any tag instead of v0.0.4
- Depending on your system it takes a few minutes to successfully build the image
Once complete, check the docker image
```
docker images
```
Create and run a Docker Container for the image
```
docker run -p 8501:8501 aitranscriber:v0.0.4
```
Note:
- docker run -p <hostport>:<8501> <container_name>:<tag_name>
- In the above command, you can play around with which port of your host system you wish to map to the 8501 port of the container
- If you used a different docker image name and/or different tag, make sure to update it in the command
Open your preferred Web Browser and navigate to http://localhost:8501
Note:
- If you used a different host port in the above command then navigate to that one, http://localhost:<host_port>
- To stop the container, in the terminal check the containter name: docker ps --all
- Now use container name with the command: docker stop <container_name>

🌏Deployment Options

Streamlit Cloud
HuggingFace Spaces
Fly
Railway
Render
Cyclic
Heroku
Digital Ocean

Google Cloud Run

Install Google Cloud CLI
Create an Account on Google Cloud
Create a New Project

Build and Push Docker Image to Google Container Registry

gcloud builds submit --tag gcr.io/<ProjectName>/<AppName>  --project=<ProjectName>

Deploy the Docker Container

gcloud run deploy --image gcr.io/<ProjectName>/<AppName> --platform managed --project=<ProjectName> --allow-unauthenticated

Amazon EC2 Instance
Azure App

(Using Google Colab/Kaggle as temporary MVP server)

pyngrok
- Step 1: Install pyngrok in Google Colab
```
! pip install pyngrok
```
- Step 2: Sign-up in ngrok and get Authentication Token
- Step 3: Authenticate
```
   from pyngrok import ngrok
   ngrok.set_auth_token("xxx")
```
- Step 4: Load the Streamlit App at port 8051, create a tunnel for it and reveal the public URL for the tunnel
```
   !nohup streamlit run app.py --server.port 8051 &
   url = ngrok.connect(8051).public_url
   print(url)
```
- Step 5: Share URL with client

localtunnel

Step 1: Install localtunnel
```
npm install -g localtunnel
```

Step 2

streamlit run Home.py & npx localtunnel --port 8501

Step 3: Share URL with client

(Using local server as temporary MVP server)

NGINX + Cloudfare/ngrok

🏗️ Future Work

Download and use audio from Youtube Video
Download and use online audio file
Use Session States and Caching for Better UX
Display the language detected propely (without using the shortcode)
Generate Dedicated SRT,VTT files for transcripts (in addition to txt)
Update Model options to honour the name of prominent Indian Scientists
Option to limit/increase input model file size
Functionality to check the validity URL provided for Youtube Video
Add Custom Favicon File
Add Scrollable Text Area for Generated Transcripts
Containerize the Application with Docker
Troubleshoot Docker Container locally
Create Basic Workflow on GitHub Actions for Docker Image Build
Create Comprehensive Workflow on GitHub Actions for Docker Image Build
Resolve bug: Youtube video with multiple audios should download default audio.
- Example: This clip from Huberman Lab is in English yet the script fetches the spanish audio codec from Youtube
Test Application by spinning up it's Container on Google Cloud Run
- Push to a particular Docker Image Registry
- Set TTL
- Play around with system resources
- Test with custom domain
Add Google Cloud's CI/CD to repo on push/pull requests
- Use cloudbuild.yaml file
- Update build time to 2 hours
Optimize Docker Image Size
Better CI/CD
Kubernetes Upgrade
Better GitHub Actions

More Features:

Burn transcripts to user-uploaded video ```python import os output_video = "final.mp4"

  os.system(f"ffmpeg -i {input_video} -vf subtitles={subtitle} {output_video}")
  ```

Summarize subtitles
Sentiment analysis on video summary
Batch transcript generation + summary + sentiment analysis
Dashboard for video review(s)

Speaker Diarization: Only if Community requires

Incorporate Speaker Diarization for Podcast/Vlog/Conversational Clips
Test it with burning transcripts to user uploaded video
Test it with transcript summarization

More Aligned Subtitles: Only if Community requires

Word Level Timestamps for transcripts + Generate ASS Transcript File
Test it with burning transcripts to user uploaded video
Test it with previous speaker diarization
Test it with transcript summarization
Improve UI Natively in Streamlit

API Development: Only if Community requires

Build API for model inference in FastAPI to handle requests asynchronously (on a different branch perhaps)
Containerize the API with Docker
Troubleshoot Docker Container for API
Host the API on Google/AWS/Linode/Heroku
Perform basic CI/CD for API
Rehost Streamlit Application on a different service (Reduce it to client side for most operations)
Play around with pyScript

Front End Development: Only if Community requires

Build Basic React Front end
Connect React Front End to FastAPI
Add Loader Animation
Add Animations for model inference times
Handling Errors in Front End/API
Upload File Component
Download Button(s)
Feedback Form
Contact Page
About Page
Home Page
Stripe Integration
Improve Navbar UI
404 Page
Footer UI
Scrollbar UI
SEO

CI/CD Pipeline (GitHub Actions)

SAST (Optional)
Kubernetes Smoke Test (Optional)
Using Super Linter for Linting (Optional)
Unit Tests (Optional)
Integration Test (Optional)

✏️ Note

To view the generated transcript file(s) in VS Code IDE install Subtitles Editor extension
To extensively edit/manipulate the generated transcript file(s) use the open source tool Subtitle Edit
For Streamlit Sharing, mentioning versions of the modules in requirements throws error at times
Large Modelv2 outperforms all other versions of Whisper in terms of performance especially in Multi-lingual Transcription. However, it takes a 10 times more V-RAM than the base model and has longer inference time
To quickly record audio from system microphone use this Python Script:
- Pre-requisities:
```
pip install pyaudio wave
```
Whisper is unable to read audio file from disk if python-ffmpeg or ffmpeg python pacakges are installed. It only works when ffmpeg-python python package is installed and not the former too
```
# Remove all ffmpeg related python packages
pip uninstall python-ffmpeg ffmpeg ffmpeg-python
# Install the appropriate pacakge for ffmpeg
pip install ffmpeg-python
```
Pixabay has a great collection of copyright free, no royalty songs that one can use for testing the application
Poor Performance for Kanada or Telegu songs (often language recognition itself fails) for base model. Example: Kantara movie's Varaha Roopam Song

Docker Container and CI/CD

Exclude as much irrelevant files as possible with .dockerignore such as README.MD, LICENSE, snapshots, notebooks, input,output,logs, etc
Minimize the number of layers (Created by RUN, COPY and ADD)
Always combine RUN apt-get update with apt-get install in the same RUN statement. Using apt-get update alone in a RUN statement causes caching issues and subsequent apt-get install instructions fail.
Using RUN apt-get update && apt-get install -y ensures your Dockerfile installs the latest package versions with no further coding or manual intervention. This technique is known as “cache busting”.
In addition, when you clean up the apt cache by removing /var/lib/apt/lists it reduces the image size, since the apt cache is not stored in a layer.
Python Docker Image Info:
- Images tagged with stretch/buster/jessie/buster/bullseye are codenames for different Debian Operating System Production releases.
- bullseye being version 11, buster being version 10, and so on. (2022)
- bookworm, trixy and forky are work-in-progress releases which may not be stable yet
- -slim - only installs the minimal packages needed to run the particular tool.
Base Image with python <= 3.9 raises issue with module backports.zoneinfoand pip fails
To build and test multi-architecture docker images locally,
- Create a new buildx instance
```
docker buildx create --use
```
- Build a new docker image for multi-architecture support
```
 docker buildx build --platform linux/arm64,linux/amd64 -t aitranscriber:multi-architecture -f Dockerfile . 
```
Checking Docker Image Build for multi-architecture is too time consuming for the current application and disabled

🛡️ License

This project is licensed under the GNU Affero General Public License v3.0 License - see the LICENSE file for details.

🙏 Acknowledgements

General Purpose Speech Recognition Model: OpenAI Whisper
- GitHub
- Paper:Robust Speech Recognition via Large-Scale Weak Supervision
Animations: LottieFiles
Favicon: PNG Repo
Sample Test Clip 1: " Thinkin About It " by Niklas Setzkorn from Pixabay

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.MD

README.MD

AI Audio Transcriber

🚀 Demo

📝 Basic Application WorkFlow

🥊CI/CD

⚒️ Set-Up Instructions

A. Without using Docker

B. Using Docker

🌏Deployment Options

🏗️ Future Work

✏️ Note

Docker Container and CI/CD

🛡️ License

🙏 Acknowledgements

Files

README.MD

Latest commit

History

README.MD

File metadata and controls

AI Audio Transcriber

🚀 Demo

📝 Basic Application WorkFlow

🥊CI/CD

⚒️ Set-Up Instructions

A. Without using Docker

B. Using Docker

🌏Deployment Options

🏗️ Future Work

✏️ Note

Docker Container and CI/CD

🛡️ License

🙏 Acknowledgements